Jonathan Millen on covert channel communication

 Jonathan Millen started work at the MITRE Corporation in 1969, after graduation from Rensselaer Polytechnic Institute with a Ph.D. in Mathematics. He retired from MITRE in 2012 as a Senior Principal in the Information Security Division. From 1997 to 2004 he enjoyed an interlude as a Senior Computer Scientist in the SRI International Computer Science Laboratory. He has given short courses at RPI Hartford, University of Bologna Summer School, ETH Zurich, and Taiwan University of Science and Technology. He organized the IEEE Computer Security Foundations Symposium (initially a workshop) in 1988, and co-founded (with S. Jajodia) the Journal of Computer Security in 1992. He has held positions as General and Program Chair of the IEEE Security and Privacy Symposium, Chair of the IEEE Computer Society Technical Committee on Security and Privacy, and associate editor of the ACM Transactions on Information and System Security.
The theme of his computer security interests is verification of formal specifications, of security kernels and cryptographic protocols. At MITRE he supported the DoD Trusted Product Evaluation Program, and later worked on the application of Trusted Platform Modules. He wrote several papers on information flow as applied to covert channel detection and measurement. His 2001 paper (with V. Shmatikov) on the Constraint Solver for protocol analysis received the SIGSAC Test of Time award in 2011. He received the ACM SIGSAC Outstanding Innovation award in 2009.

Luke Muehlhauser: Since you were a relatively early researcher in the field of covert channel communication, I’d like to ask you about the field’s early days, which are usually said to have begun with Lampson (1973). Do you know when the first covert channel attack was uncovered “in the wild”? My impression is that Lampson identified the general problem a couple decades before it was noticed being exploited in the wild; is that right?

Jonathan Millen: We might never know when real covert channel attacks were first noticed, or when they first occurred. When information is stolen by covert channel, the original data is still in place, so the theft can go unnoticed. Even if an attack is discovered, the victims are as reluctant as the perpetrators to acknowledge it. This is certainly the case with classified information, since a known attack is often classified higher than the information it compromises. The only evidence I have of real attacks before 1999 is from Robert Morris (senior), a pioneer in UNIX security, and for a while the Chief Scientist of the National Computer Security Center, which was organizationally within NSA. He stated at a security workshop that there had been real attacks. He wouldn’t say anything more; it was probably difficult enough to get clearance for that much.

Luke: From your own perspective, what are some of the most interesting developments in covert channel communication since you wrote “20 Years of Covert Channel Modeling and Analysis” in 1999?

Jonathan: That’s a tough question, for several reasons. Like many research areas that have been in existence for many years, covert channel research has splintered into several specialties. Some of the subtopics are noninterference models, language-based information control via type theory, steganography, and so on.
While specialization is necessary for deep progress, I wish there were more attention given to the pragmatic “so what” questions about covert channels. For example, information flow security is often defined in terms of noninterference, which can, unfortunately, be defined in several subtly different ways. All of them rely on some form of behavioral equivalence, a strong condition guaranteeing that an unauthorized observer cannot tell when a more privileged user is present. Even one bit, or a fraction of a bit, of information about the protected party’s data or actions is considered too much. Yet, what we really want to know is whether a recurrent channel can be set up, yielding a large or unbounded amount of information. Furthermore, information in the Shannon sense may or may not be useful. One paper that attracted my attention is “Quantifying information flow with beliefs”, by Clarkson, Myers, and Schneider, 2009, because it dealt with the accuracy of information.
Another pragmatic issue is the trustworthiness of the system functionality. In general, I am distrustful of security approaches that depend on access control software or the design of the language in which it is written. Even the firmware and hardware below the operating system kernel is suspect. In the last few years, my main research interest was in trusted computing, as supported by trusted platform modules, with their capability — available but largely unused in modern platforms — to cryptographically check the integrity of system software by a bootstrap sequence starting with the tamper-resistant TPM. And still, having done that, we know only that the system software is genuine, not that it is correct or free from covert channels. It is encouraging that work in software and hardware verification continues, but commonly used systems are still beyond its reach.
Complexity is the enemy of security. The more you have to depend on, the less likely it is that you can understand it well enough to exclude vulnerabilities. A couple of corollaries: the more complex your security policy is, the less robust it is, because more complex software is needed for it. And the more specialized a system component is to support security, the more attractive it is as a target for an attack. I suppose that the optimistic view of this is that if we use a typically complex modern system, covert channels are the least of our worries!

Luke: From your many decades of experience in computer security, do you know of cases where someone was worried about a computer security or safety challenge that wasn’t imminent but maybe one or two decades away, and they decided to start doing research to prepare for that challenge anyway — e.g. perhaps because they expected the solution would require a decade or two of “serial” research and/or engineering work, with each piece building on the ones before it, and they wanted to be prepared to meet the challenge near when it arrived? Lampson’s early identification of the “confinement problem” looks to me like it might be one such example, but maybe I’m misreading the history there.

Jonathan: In computer security, what usually happens is that someone realizes that a vulnerability already exists, but it is not clear how long it will take for a malicious party to take advantage of it. Another factor delaying the onset of relevant research and development is that long-term efforts aimed at removing vulnerabilities are risky and their results may be inadequate or eclipsed by different problems in the future.
If the difficulty is social, such as getting standards updated, the difficulty is real, but the basic engineering knowledge is mostly available early on. For example, the TCP sequence number attack for session hijacking was pointed out by Morris in 1985, taken advantage of by Mitnick in 1995, led to Bellovin’s standards-related recommendations in 1996, and only recently has the availability of encryption-based authentication and IPv6 brought a widespread solution within reach.
If the difficulty is in the engineering, such as the development of security kernels and mandatory access policies to defeat Trojan horses, or the installation and use of TPMs, there is an industry to be moved, one driven by the need for commercial viability of new products. And after all that effort, some vulnerabilities remain.
Computer scientists can help by developing powerful general-purpose tools and techniques with fundamental computer science goals — tools that are not problem-dependent. By this, I mean such things as verification tools and model checkers, as well as encryption algorithms and applications. Such tools can ease the engineering burden when new problems and design ideas emerge. Development of these tools deserve decades of research building on prior work, and those who are capable of conducting this activity successfully should be encouraged and supported.

Luke: You mention verification tools and model checkers. I was recently speaking to a significant figure in the computer security and safety community (I don’t have permission to give her name) about the common claim that formal methods can catch corner bugs which are unlikely to be caught by testing alone. She mentioned that if the formal methods community has ever subjected formal methods to careful experimentation to demonstrate that verification really does catch bugs that aren’t likely to be found by testing, she wasn’t aware of it. Are you aware of any such demonstrations? If not, how would you describe the value in formal methods for safety and security purposes?

Jonathan: It seems unfair to me to ask for careful experimentation to justify formal methods in comparison to testing. One reason for this that was pointed out when formal methods were being recommended for security kernel verification, is that very few security kernels are actually developed, so any experimental results are not going to be statistically significant.
One experiment that was done in the very beginning of the history of formal specifications was by David Parnas, credited by some with inventing the notion of “formal nonprocedural specification.”1 In fact, many of us referred to such specifications at first as “Parnas specifications”, but he let it be known (through Peter Neumann) that he disapproved of that. Parnas’ experiment was a small classroom experiment designed to investigate, not the efficacy of verification of formal specifications, but rather the plausible idea that specifications determine implementations. Parnas’ classroom experiment, in which several groups were given the same specification to implement, disproved that (or at least cast doubt on that) because different groups implemented their specified task in very different ways, despite the very detailed constraints on input-output behavior in the specification. (I heard the experiment described at a conference presentation by Parnas, but I can’t find the paper or the citation now.)
This result turned out to be useful to me a few years later when DoD sponsors argued that analysis of the formal specifications of security kernels should be classified higher than the programs themselves, even when the analysis proved the absence of access control violations and storage channels, because the specifications allowed readers to deduce features of the implementation that would tell them about timing and other channels. I pointed out that Parnas had given evidence that their fears were not justified.
One indicator of the value of formal methods, model checking in particular, is the fact that it has been used routinely in the design of CPUs, which have rather complicated internal strategies for managing memory and register caching. (One citation I found with a quick Google to confirm this is “Fifteen years of formal property verification in Intel”, by Limor Fix, of Intel Research Pittsburgh, published in a 2008 Springer-Verlag book, 25 Years of Model Checking.)

Luke: You also wrote that “Computer scientists can help by developing powerful general-purpose tools and techniques… [e.g.] verification tools and model checkers, as well as encryption algorithms and applications… Development of these tools deserve decades of research building on prior work, and those who are capable of conducting this activity successfully should be encouraged and supported.”
What are some specific examples of tools for software system safety or security that you wish were receiving more development talent and funding than is currently the case? E.g. probabilistic model-checkers, or verified libraries for program synthesis, or whatever you think is most urgent or underfunded.

Jonathan: My gut reaction is, all of the above. When it comes to specific tools, each of us has preferences based on what we needed, what we have used, and what is most accessible. When I was at SRI International, I used the PVS theorem prover, and later, SRI’s SAL environment, which hosts several flavors of model checking algorithms. Both of those have benefited from the highest levels of talent; the challenge is that the job is never finished, especially if it is a thriving, well-used system. Funding sources typically emphasize urgent applications or brand new ideas, as they should, but they tend to neglect the steady, unromantic work needed to keep a software suite up to date with respect to the evolution of hardware, operating systems, algorithmic advances, and so on. And tools that support formal methods do not have the same opportunities for income as Windows, Apple’s OS X, and other commercial systems, even with a strong user community.
Incidentally, although I have emphasized general-purpose tools, that does not mean that I am against tools designed for particular applications such as security. After all, I spent a fair amount of time on a protocol security analyzer (called the Constraint Solver). Specialized tools, however, benefit in brevity and readability from being built on top of an expressive language with powerful primitives — for my application, Prolog, specifically SWI-Prolog.
A word of caution, though: it’s one thing to build an analysis tool with a complex system under it, and another to build a supposedly secure system that way. A lesson learned from studying covert channels is that the whole system under the secure interface is a potential source of vulnerabilities. So how can we build secure systems? The main requirement, I think, is simple: don’t let the enemy program your computer (the computer holding your data). But how can we prevent that? That’s the hard part.

Luke: Thanks, Jonathan!
Parnas (1972).The post Jonathan Millen on covert channel communication appeared first on Machine Intelligence Research Institute.


Wolf Kohn on hybrid systems control

 Dr. Wolf Kohn is the Chief Scientist at Atigeo, LLC, and a Research Professor in Industrial and Systems Engineering at the University of Washington. He is the founder and co-founder of two successful start-up companies: Clearsight Systems, Corp., and Kohn-Nerode, Inc. Both companies explore applications in the areas of advanced optimal control, rule-based optimization, and quantum hybrid control applied to enterprise problems and nano-material shaping control. Prof. Kohn, with Prof. Nerode of Cornell, established theories and algorithms that initiated the field of hybrid systems. Prof. Kohn has a Ph.D. in Electrical Engineering and Computer Science from MIT, at the Laboratory of Information and Decision Systems. Dr. Kohn is the author or coauthor of over 100 referred papes, 6 book chapters and with Nerode and Zabinsky has written a book in Distributed Cooperative inferencing. Dr. Kohn Holds 10 US and international patents.

Luke Muehlhauser: You co-founded the field of hybrid systems control with Anil Nerode. Anil gave his impressions of the seminal 1990 Pacifica meeting here. What were your own impressions of how that meeting developed? Is there anything in particular you’d like to add to Anil’s account?

Wolf Kohn: The discussion on the first day of the conference centered on the problem of how to incorporate heterogeneous descriptions of complex dynamical systems into a common representation for designing large scale automation. What came almost immediately were observations from Colonel Mettala and others that established as a goal the finding of alternatives to classic approaches based on combining expert systems with conventional control and system identification techniques.
These approaches did not lead to robust designs. More important, they did not lead to a theory for the systematic treatment of the systems DOD was deploying at the time. I was working on control architectures based on constraints defined by rules, so after intense discussions among the participants Nerode and I moved to a corner and came up with a proposal to amalgamate models by extending the concepts of automata theory and optimal control to characterize the evolution of complex dynamical systems in a manifold in which the topology was defined by rules of operation, and behavior constraints and trajectories were generated by variational methods. This was the beginning of what we would be defined later on as “hybrid systems.”

Luke: Which commercial or governmental projects would you name as being among the most significant success stories of the hybrid systems research program, from 1990 to the present day?

Wolf: There are many applications today that use hybrid systems as the basic technology. These are a few of the ones I am I personally familiar with:

A demand forecaster and an inventory control and management system being deployed by the Microsoft Dynamics group.
A battlefield simulator deployed by the Army’s Piccatiny Arsenal.
A generic people and resource scheduling system deployed by Clearsight Systems.
A cooperative distributed inference system deployed by Atigeo with applications on medical informatics and smart electric power network management systems.
A quantum hybrid control system for capturing and storing sunlight being prototyped by Kohn-Nerode LLC.

Luke: What are the new theoretical developments in hybrid systems of the past 10 years that are most impressive or interesting to you? What kinds of advances do you think we might see in the next 10 years?

Wolf: For me the most important advances in hybrid systems are in three areas:

Representation: We have found that the behavior of dynamical systems characterized by multiple heterogeneous models can be effectively characterized by Hamiltonian functions. The interaction with multiple models and the transfer of information from one model to another is defined by the interaction of Hamiltonian forms. This fact allows hybrid systems to be a preferred theory and implementation technology for the development of new control approaches such as mean field agent-based distributed control and gauge theory and most importantly, control and optimization specified and implemented by rules.
Hybrid systems control design, which allows for the specification of architectural structures and the control requirements as part of the formulation of a control problem. This fuses the physical and empirical data about the dynamic process to be controlled. The control specification and the computational requirements are achieved because control performance specifications and architectural constraints are models of the controlled process.
Agent-based distributed control: Nerode and I, both together and separately have developed a variational theory, based on hybrid systems, for dynamic synchronization of multiple agents participating in the control of a distributed process. The variational theory generalizes a principle in network theory, called Tellegen’s Theorem. The theory provides for active synchronization with no umpire for network interaction between the agents. A version of this theory has been used to implement an agent-based architecture for control, uncertainty management, and learning in several applications. This architecture is known by the acronym MAHCA (Multiple Agent Hybrid Control Architecture).
Metacontrol: This is a theory for controlling the performance and behavior of implemented algorithms. The dynamics in this case is a computational multitask process. The objective is to make it run faster with less memory and with active synchronization between tasks. We are developing a computational hybrid systems theory based on metacontrol. Preliminary implementations of this theory on optimization algorithms has shown very promising results in terms of reduction of compute time, real time synchronization and rule based optimization.

Luke: I’m particularly interested in the safety challenges presented by the increasingly autonomous AI systems of the future. Self-driving cars are on their way, the U.S. military is working toward autonomous battlefield robots of various types, etc. Do you think hybrid systems control, and relatively modest extensions to it, will be sufficient to gain high assurances of the safety for the more-capable autonomous systems we’re likely to have in 10 or 20 years, or do you think other contemporary control and verification approaches have a better shot at addressing that problem, or do you think entirely new approaches will need to be developed?

Wolf: I will break down my answer in two parts: (1) the autonomy issue and (2) the verification issue.
Autonomy: This was one of the central questions brought up by the initial sponsors of hybrid systems (ARPA, NIST, ARO, SAP). Our answer early on came with the following proposition which we implemented in a battlefield dynamics simulator: given a process to be controlled, build a model of desired behavior based on the performance specifications, regulations, and economic operation, and construct the controlled process dynamics. This is a hybrid system; let’s call it S.
Then we build a model, another hybrid system, say C, representing specified safety rules and constraints and hybridize S and C to produce a new Hybrid control system S1. This approach is only successful in semi-closed, quasi-stationary systems.
For the large scale autonomous systems, one needs to allow the hybrid system controller to detect, learn and dispose of un-programmed situations. To do this, a class of hybrid controllers (known as agents in our papers) contain directives that implement Learning from Sensory Data. One example is what we might call “Learning by Failure Predictor and Repair” systems. This particular system operates as follows: a safety agent (or agents) is (are) monitoring the controlled system operation, and infer(s) whether it is operating in a feasible region and what is the likelihood that it is going to leave the feasibility region (a Failure) in the near future (“near” is a concept depending on the controlled system). This likelihood determines the response to the failure: a Repair operation.
Then, another agent application designs a controller that implements the Repair operation. Note this is not necessarily an adaptation of an existing controller. We use the fact that the design procedure of hybrid controllers is itself a hybrid system defining the synthesis approach. We term the resulting design procedure a repair hybrid control. We have used this approach in a prototype microgrid management and control system that is near deployment with considerable success: i.e., high robustness, resiliency, recovery: all this while maintaining good performance.
Another element that agent-based hybrid systems provide is the ability to improve safety via redundancy.
In short, my answer to the first part of the question may be summarized as follows:
Hybrid systems is a first principles platform that allows for the incorporation of safety, learning and repair. What I believe the research in this area should be focused on is how to populate the system with information about the application and to allow for heuristics and empirical constraints and rules, and to provide the structural Failure and Repair mechanisms I outlined above. This approach builds on top on existing control techniques but incorporates a new concept of structural adaptation that is essential for the level of autonomy posed in your question.
Verification: many researchers have proposed methods for hybrid systems validation and verification prior to deployment. In our approach we are happy to use some of these techniques. Our contribution in this area called for verification of online designs, as discussed above. So, we are developing hybrid systems to model verification principles, with the idea to amalgamate them in near real time to our applications.

Luke: Do you think our capacity to make systems more autonomous and capable will outpace our capacity to achieve confident safety assurances for those systems?

Wolf: I believe we have made great progress on increasing autonomy in most of the systems we are designing today. I also believe we have paid far less attention to develop the methods, sensory redundancy principles, and theory of design for safety performance.
Nerode and I are working on developing an advanced magnetic battery using quantum hybrid control methodology. We found that the key ingredients of this battery for it to operate safely are principles of safety that have been used in non-autonomous systems for the last 100 years. We are encoding these principles formally as part of our design algorithms. Perhaps this way may be generalized to obtain acceptable levels of safety.

Luke: Thanks, Wolf!
The post Wolf Kohn on hybrid systems control appeared first on Machine Intelligence Research Institute.

Wolf Kohn on hybrid systems control


MIRI’s April 2014 Newsletter

Research Updates

New technical report: “Botworld 1.0.”
9 new expert interviews: Max Tegmark on the mathematical universe, Randal Koene on whole brain emulation, Michael Carbin on integrity properties in approximate computing, Anil Nerode on hybrid systems control, Lyle Ungar on forecasting, Erik DeBenedictis on supercomputing, Will MacAskill on normative uncertainty, Diana Spears on the safety of adaptive agents, and Paulo Tabuada on program synthesis for cyber-physical systems.

News Updates

We are actively hiring for three positions: math researcher, science writer, and executive assistant. Compensation is competitive and visa are possible.
We’re also accepting applications on a rolling basis for future MIRI research workshops. Please apply if you’re a curious academic looking for exposure to our material. If you’re accepted, we’ll contact you about potential workshops over time, as we schedule them.
We published our 2013 in Review: Fundraising post.
Louie was interviewed at io9 about the unsuitability of Asimov’s Three Laws of Robotics for machine ethics.
Luke was interviewed at The Information about MIRI’s research. Unfortunately, the interviewer’s editing introduced several errors: see here.

Other Updates

April 11th is your last chance to vote for Nick Bostrom and others in Prospect‘s World Thinkers 2014 poll.
Oxford University is hosting an effective altruism conference in July. Speakers include Nick Bostrom and Derek Parfit. Register here.
Registration is also open for Effective Altruism Summit 2014, in August in the Bay Area.
Our frequent collaborators at the Future of Humanity Institute are currently hiring a research fellow in philosophy, for a focus on population ethics.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI’s April 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s April 2014 Newsletter


New Report: Botworld

Today MIRI releases a new technical report: “Botworld 1.0” (pdf) by recent hires Nate Soares and Benja Fallenstein. The report is a “literate” Haskell file, available from MIRI’s Github page.
Soares explains the report on his accompanying Less Wrong post, which is also the preferred place for discussion of the report:
This report introduces Botworld, a cellular automaton that provides a toy environment for studying self-modifying agents.
The traditional agent framework, used for example in Markov Decision Processes and in Marcus Hutter’s universal agent AIXI, splits the universe into an agent and an environment, which interact only via discrete input and output channels.
Such formalisms are perhaps ill-suited for real self-modifying agents, which are embedded within their environments. Indeed, the agent/environment separation is somewhat reminiscent of Cartesian dualism: any agent using this framework to reason about the world does not model itself as part of its environment. For example, such an agent would be unable to understand the concept of the environment interfering with its internal computations, e.g. by inducing errors in the agent’s RAM through heat.
Intuitively, this separation does not seem to be a fatal flaw, but merely a tool for simplifying the discussion. We should be able to remove this “Cartesian” assumption from formal models of intelligence. However, the concrete non-Cartesian models that have been proposed (such as Orseau and Ring’s formalism for space-time embedded intelligence, Vladimir Slepnev’s models of updateless decision theory, and Yudkowsky and Herreshoff’s tiling agents) depart significantly from their Cartesian counterparts.
Botworld is a toy example of the type of universe that these formalisms are designed to reason about: it provides a concrete world containing agents (“robots”) whose internal computations are a part of the environment, and allows us to study what happens when the Cartesian barrier between an agent and its environment breaks down. Botworld allows us to write decision problems where the Cartesian barrier is relevant, program actual agents, and run the system.
As it turns out, many interesting problems arise when agents are embedded in their environment. For example, agents whose source code is readable may be subjected to Newcomb-like problems by entities that simulate the agent and choose their actions accordingly.
Furthermore, certain obstacles to self-reference arise when non-Cartesian agents attempt to achieve confidence in their future actions. Some of these issues are raised by Yudkowsky and Herreshoff; Botworld gives us a concrete environment in which we can examine them.
One of the primary benefits of Botworld is concreteness: when working with abstract problems of self-reference, it is often very useful to see a concrete decision problem (“game”) in a fully specified world that directly exhibits the obstacle under consideration. Botworld makes it easier to visualize these obstacles.
Conversely, Botworld also makes it easier to visualize suggested agent architectures, which in turn makes it easier to visualize potential problems and probe the architecture for edge cases.
Finally, Botworld is a tool for communicating. It is our hope that Botworld will help others understand the varying formalisms for self-modifying agents by giving them a concrete way to visualize such architectures being implemented. Furthermore, Botworld gives us a concrete way to illustrate various obstacles, by implementing Botworld games in which the obstacles arise.
Botworld has helped us gain a deeper understanding of varying formalisms for self-modifying agents and the obstacles they face. It is our hope that Botworld will help others more concretely understand these issues as well.
The post New Report: Botworld appeared first on Machine Intelligence Research Institute.

New Report: Botworld


Paulo Tabuada on program synthesis for cyber-physical systems

 Paulo Tabuada was born in Lisbon, Portugal, one year after the Carnation Revolution. He received his “Licenciatura” degree in Aerospace Engineering from Instituto Superior Tecnico, Lisbon, Portugal in 1998 and his Ph.D. degree in Electrical and Computer Engineering in 2002 from the Institute for Systems and Robotics, a private research institute associated with Instituto Superior Tecnico. Between January 2002 and July 2003 he was a postdoctoral researcher at the University of Pennsylvania. After spending three years at the University of Notre Dame, as an Assistant Professor, he joined the Electrical Engineering Department at the University of California, Los Angeles, where he established and directs the Cyber-Physical Systems Laboratory.
Paulo Tabuada’s contributions to cyber-physical systems have been recognized by multiple awards including the NSF CAREER award in 2005, the Donald P. Eckman award in 2009 and the George S. Axelby award in 2011. In 2009 he co-chaired the International Conference Hybrid Systems: Computation and Control (HSCC’09) and in 2012 he was program co-chair for the 3rd IFAC Workshop on Distributed Estimation and Control in Networked Systems (NecSys’12). He also served on the editorial board of the IEEE Embedded Systems Letters and the IEEE Transactions on Automatic Control. His latest book, on verification and control of hybrid systems, was published by Springer in 2009.

Luke Muehlhauser: In “Abstracting and Refining Robustness for Cyber-Physical Systems,” you and your co-author write that:
…we present a design methodology for robust cyber-physical systems (CPS) [which]… captures two intuitive aims of a robust design: bounded disturbances have bounded consequences and the effect of sporadic disturbances disappears as time progresses.
You use an “abstraction and refinement” procedure for this. How does an abstraction and refinement procedure work, in this context?

Paulo Tabuada: Cyber-physical systems are notoriously difficult to design and verify because of the complex interactions between the cyber and the physical components. Although control theorists have developed powerful techniques for designing and analyzing physical components, say described by differential equations, and computer scientists have developed powerful techniques for designing and analyzing cyber components, say described by finite-state models, these techniques are for the most part incompatible. The latter rely on discrete mathematics while the former rely on continuous mathematics. Our approach is based on replacing all the physical components by cyber abstractions so that all the remaining design and verification tasks can be done in the cyber world. The construction of these abstractions is based on rigorous numerical simulation combined with an analysis of the differential equation models to guarantee that the original physical components and its abstractions are equivalent up to a desired precision. Technically, “equivalent up to a desired precision” means approximately bisimilar and intuitively this means that both models generate the same set of behaviors up to a desired precision.

Luke: Last summer you gave a four-part tutorial (p1, p2, p3, p4) on program synthesis for cyber-physical systems. For someone who isn’t familiar with program synthesis, can you describe how it’s done in the context of cyber-physical systems, and give an example of such a system that has been implemented?

Paulo: Program synthesis is already a challenging problem for software only systems. In the context of cyber-physical systems it becomes even more challenging since the objective is to synthesize a program that will make a physical system behave as intended. I am currently working on two projects related to program synthesis. One of these has for objective the synthesization programs that control the behavior of bipedal robots. The very same techniques are being used in the same project to synthesize programs for adaptive cruise control and lane departure control systems in collaboration with Toyota and Ford. To give you an idea of where the challenge lies, let me recall how this project started. My colleagues Jessy Grizzle at UMich and Aaron Ames at Texas A&M came to me and shared their frustration with the programs they developed to control their robots. While they were very satisfied making their robots walk on flat and unobstructed surfaces, they faced large problem on uneven terrain. Their approach was to develop a set of rules, i.e., a reactive program that responds to the stimuli provided by the sensors by determining which actuators should be used and how. Although the rules were developed based on common sense, it soon became clear it was impossible to predict how the execution of these rules would impact the motion of the robot. Moreover, a small change in the rules would lead to completely different and unexpected behavior. Our approach is to construct a finite-state abstraction of the robot dynamics and then to synthesize a reactive program that forces this abstraction to satisfy a desired specification. The synthesis of these reactive programs is done via the solution of a two player game where the synthesized program plays against the robot’s environment and enforces the specification no matter which action the environment takes. In this way, rather than verifying a reactive program, we synthesize one that is guaranteed to be correct by construction.

Luke: Robots and other AI systems are becoming increasingly autonomous in operation: self-driving cars, robots that navigate disaster sites, HFT programs that trade stocks quickly enough to “flash crash” the market or nearly bankrupt a large equities trader, etc.
How might current AI safety methods (formal verification, program synthesis, simplex architectures, etc.) be scaled up to meet the safety challenges raised for highly autonomous systems operating in unknown, continuous, dynamic environments? Will our capacity to make systems more autonomous and capable outpace our capacity to achieve confident safety assurances for those systems?

Paulo: Indeed, that is what has been happening so far. Our ability to create large complex systems is far ahead of our understanding of the basic scientific principles ensuring their safe operation. In my opinion, synthesis based approaches have the best chance to scale to the level of existing applications. Formal verification of these systems is extremely hard since the design space is very very large. No formal verification technique can handle the wide variety of systems from such large design space. When using synthesis, however, we can reduce the design space and thus obtain much more structured designs for which formal guarantees can be obtained. What we loose, in turn, is the ability to find all the design solutions. This tradeoff is well understood in other areas of engineering, e.g., brick and mortar or wood framed building construction. I believe that it wont take long until we discover a few synthesis techniques that guarantee safety and correctness of highly autonomous systems, even if these result in somewhat more conservative solutions.

Luke: You write that “it wont take long until we discover a few synthesis techniques that guarantee safety and correctness of highly autonomous systems, even if these result in somewhat more conservative solutions.” Could you clarify what you have in mind? For example, what might a “conservative solution” look like for a high-assurance program synthesis solution for the software of a self-driving car? (Feel free to give a different example if you prefer.)

Paulo: By conservative I mean that we will have to restrict the synthesized software so that it falls within a class for which synthesis techniques are available. A simple example is software that can be described by timed-automata. Although it might be convenient to make a decision based on a generic function of several timers, to stay within the timed-automata class we are only allowed to compare timers against constants or compare differences between timers against constants. For a self-driving car, this could mean taking decisions based on delays exceeding thresholds but not being allowed to make decisions based on the average of several delays.

Luke: Thanks, Paulo!
The post Paulo Tabuada on program synthesis for cyber-physical systems appeared first on Machine Intelligence Research Institute.


Diana Spears on the safety of adaptive agents

 Diana Spears is an Owner and Research Scientist at Swarmotics, LLC. Previously, she worked at US government laboratories (Goddard, NIST, NRL) and afterwards she was an Associate Professor of Computer Science at the University of Wyoming. She received both the MS and PhD (1990) degrees in Computer Science from the University of Maryland, College Park.
Dr. Spears’s research interests include machine learning, adaptive swarm robotic sensing networks/grids, computational fluid dynamics based algorithms for multi-robot chemical/biological plume tracing and plume mapping, and behaviorally-assured adaptive and machine learning systems. Dr. Spears pioneered the field of safe adaptive agents with her award-winning (2001 NRL Alan Berman Research Publication Award) publication entitled, “Asimovian adaptive agents.” Most recently she and her husband co-edited the book “Physicomimetics: Physics-Based Swarm Intelligence,” published by Springer-Verlag in 2012.

Luke Muehlhauser: In Spears (2006) and other publications, you’ve discussed the challenge of ensuring the safety of adaptive (learning) agents:
a designer cannot possibly foresee all circumstances that will be encountered by the agent. Therefore, in addition to supplying an agent with a plan, it is essential to also enable the agent to learn and modify its plan to adapt to unforeseen circumstances. The introduction of learning, however, often makes the agent’s behavior significantly harder to predict. The goal of this research is to verify the behavior of adaptive agents. In particular, our objective is to develop efficient methods for determining whether the behavior of learning agents remains within the bounds of prespecified constraints… after learning…
…Our results include proofs that… with respect to important classes of properties… if the [safety] property holds for the agent’s plan prior to learning, then it is guaranteed to still hold after learning. If an agent uses these “safe” learning operators, it will be guaranteed to preserve the properties with no reverification required. This is the best one could hope for in an online situation where rapid response time is critical. For other learning operators and property classes our a priori results are negative. However, for these cases we have developed incremental reverification algorithms that can save time over total reverification from scratch.
What do you mean by “incremental” reverification algorithms, as in the last sentence I quoted?

Diana Spears: Verification (model checking, in particular) consists of proving that a computer program satisfies a desirable property/constraint and, if it does not, a counterexample is provided. In my work, I assume that this program is a (multi)agent plan for action.  In most real-world applications, plans are typically enormous and therefore verification may be quite time-consuming. Suppose the safety property/constraint that agent A must always obey is that “agent A should always be at least M units away from agent B” (to prevent collisions). Let’s assume that initial verification proved that the entire plan (consisting of all action sequences that agent A could ever possibly take) is guaranteed to obey this property in all circumstances. Furthermore, let’s assume that adaptation is required after the agent has been fielded, where adaptation consists of applying a machine learning operator to modify the plan. For example, suppose a specific part of the plan states that agent A should “move from location 3 to location 4 if there is a clear path between 3 and 4, the ground is fairly level (e.g., nowhere higher than X or lower than Y between locations 3 and 4), and if the schedule permits such movement at this time.” Then an example machine learning operator might change the “4” to “6” based on new information about the task being performed.
Note that this learning operator only modifies one condition in one miniscule portion of the entire plan.Therefore, why re-verify that the entire plan still satisfies the desired property after learning? Why not re-verify only the specific portion of the plan that was modified, as well as any parts of the plan that depend on the modified portion?  That is what “incremental re-verification” does. It localizes post-adaptation verification to only those parts of the plan that are essential to re-verify.  In doing so, it improves the time complexity of re-verification. Time complexity is a very important and practical consideration for online systems – especially those that operate in real-time or in time-critical situations. In my research I ran numerous experiments comparing the CPU time of re-verifying an entire plan versus localized incremental re-verification of the plan after learning. My results showed as much as a 1/2-billion-fold speedup using incremental re-verification! And that’s using a plan that’s tiny compared to what agents would typically use in the real world.

Luke: With what kinds of agent programs have you explored this issue? What do the agents do, in what environment, and what kinds of safety properties do you prove about them?

Diana: Because of the strong relevance of the topic “safe adaptation” to aerospace applications, I chose to focus on NASA-related (multi-)agent programs. I depict a scenario in which a spacecraft has landed on another planet, such as Mars, and from which multiple mobile rovers emerge. The plans (programs) for the spacecraft  lander, as well as the planetary rovers, are for collecting, retrieving and transmitting/delivering data and/or samples from the surface of the planet. I prove “safety” and “liveness” properties. An example of “safety” is, “It is always the case that agent R is not delivering at the same time that agent L is transmitting.” Here, L is the lander and R is one of the rovers. This property/constraint prevents problems that may arise from the lander simultaneously receiving new data from R while transmitting older data to Earth. An example of “liveness” is, “If agent R executes the ‘deliver’ action, then eventually agent L will execute the ‘receive’ action.”

Luke: Have you or anyone else built on this specific line of work since 2006? What are some natural “next steps” for this particular line of research?

Diana: There were essentially three main offshoots of my research that I’m aware of – from NASA Ames, SRI, and USC. I’ll start with the NASA Ames offshoot. Around the year 2000 I gave a talk about “Asimovian Adaptive Agents” at NASA Ames. My instincts about the strong relevance of this work to NASA, and more generally aerospace, proved to be correct. (Furthermore, it appears to be highly relevant to any automated transportation, including automated cars/highways.) The researchers at NASA Ames rapidly and eagerly followed up on my talk with a flurry of related work, including research and publications. These researchers focused on “reference models,” which are used for online, run-time I/O checking. Instead of using temporal logic properties to verify, they used control theoretic properties such as “stability” and “performance.” Perkins and Barto also used Lyapunov stability as the property of interest when building adaptive learning agents1. For examples of the NASA Ames research and other related work, see the papers that appeared in the NIPS’04 workshop on “Verification, validation, and testing of learning systems”2. Dietterich gave a follow-up talk at NIPS’06 on this topic3. The NASA Ames offshoot has continued to be active post-2006, as exemplified by the many recent contributing papers in Schumann’s 2010 book4. Furthermore, Vahram Stepanyan and others at NASA Ames have been working on a project called “Integrated Resilient Aircraft Control” (IRAC), whose goal is to validate multidisciplinary integrated aircraft control design tools and techniques that will enable safe flight despite unexpected adverse conditions5.
Shortly after my Ames talk, a second offshoot was initiated by John Rushby at SRI International. The SRI follow-on research continued to focus on formal methods with model checking, which is what I had originally worked with. However more recently this work has moved in a more similar direction to that of Ames6. For example, in this paper Rushby introduces the idea of using a “safety case” that leads to an online performance monitor. And even more recently, Ashish Tiwari at SRI has worked on bounded verification of adaptive neural networks in the context of the IRAC project7.
Next, consider a third offshoot. This is the research at the University of Southern California (USC) by Milind Tambe and others. These USC researchers built on my specific line of work, but they decided to address the important issue of mixed-initiative situations (also called “adjustable autonomy”), where humans and artificial agents collaborate to achieve a goal. Their multiagent plans are in the form of Partially Observable Markov Decision Processes (POMDPs) and they check safety constraints in this context. The first paper of theirs that I’m aware of on the topic of Asimovian adaptive (multi)agents appeared in 20068. In 2007, Nathan Schurr got his Ph.D. on this topic9. Milind Tambe continues to teach a very popular courses on “Artificial Intelligence and Science Fiction” in which he discusses his research on Asimovian multiagents.
Finally, I’ll mention miscellaneous post-2006 research that continues to build on my earlier line of work. For one, during 2006-2008 I was part of a DARPA Integrated Learning initiative that focused on methods for airspace deconfliction. Two of my graduate students, Antons Rebguns and Derek Green, along with Geoffrey Levine (U of Illinois) and Ugur Kuter (U of Maryland), applied safety constraints to planners10. Their work was inspired by my earlier research on Asimovian agents. There are also researchers currently building on the NASA Ames work: an international group11, Zhang and other researchers at Michigan State University12, and Italian researchers who are building on Zhang’s work13. Also, Musliner and Pelican (Honeywell Labs), along with Goldman (SIFT, LLC) started building on my incremental re-verification work in particular – back in 200514, and apparently Musliner is still doing verification and validation (V&V) of adaptive systems.
Now I’ll respond to your second question about natural “next steps” for this particular line of research. I believe that all of the research mentioned above is exciting and shows promise. But I want to particularly emphasize the NASA/SRI direction as potentially fruitful for the future. This is based on my personal experiences with machine learning, formal methods, V&V, and AI in general. I have always found that formal methods and other logic-based approaches are, for computational reasons, difficult to scale up to complex real-world problems. Over the course of my career, I have leaned more towards stochastic methods for machine learning, and run-time checking for V&V. Therefore I hope that the aerospace researchers will continue in the directions they have adopted. However I also believe that they should widen their horizons. There are numerous techniques currently available for runtime monitoring and checking, e.g., see the online software self-checking, self-testing, and self-correcting methods of Ronitt Rubinfeld15, or the run-time monitoring and checking of Insup Lee and Oleg Sokolsky16. I believe it would be potentially very fruitful to explore how many of the available monitoring and checking techniques are applicable to the behavioral assurance of adaptive systems.
But, most importantly, there is a topic that is critical to the future of building trustworthy adaptive systems and needs to be explored in great depth. That’s the issue of self-recovery/repair. Since around 1998-1999, my colleagues and I have been addressing self-repair in the context of swarm robotics1718. Our research focuses primarily on physics-based approaches to controlling swarm robotic formations – because physics naturally obeys the “principle of least action,” i.e., if a formation is disturbed then it will automatically perform the minimal actions required to repair the disturbance. This repair is locally optimal but is not necessarily globally optimal. In other words, we have dropped the requirement of global optimality, focusing on “satisficing” behavior instead.  Organic and natural physical systems are not perfect, but their lack of perfection often makes them more robust. There are systems where we need precise guarantees of behavior (e.g., the dynamic control of an airplane wing, to ensure that the plane does not stall and crash). But for other tasks, perfection and optimality are not even relevant (e.g., the Internet). We have demonstrated the feasibility of our research both in simulation and on real robots on numerous tasks, including uniform coverage, chain formations, surveillance, the movement of formations through environments with obstacles, and chemical source localization19. Hopefully other researchers will also explore physics-based systems. I believe that an excellent “safe adaptive (multi)agent” architecture would consist of a physics-based controller at the lower level, with a monitor/checker at a higher level to provide strict behavioral guarantees when needed. In particular, a more sophisticated version of our original design in [17] would be quite promising.
In summary, I sincerely hope that the above-mentioned research will continue in the fruitful directions it has already taken, and I also hope that students and researchers will pursue additional, novel research along these lines. It seems to me that the topic of “safe adaptation” is “low-hanging fruit.” DARPA20 and other funding agencies have expressed to me their desire to fund research on this topic – because it must be satisfactorily addressed if we are to have deployable, adaptive systems that we can trust.

Luke: In the lines of work you outlined above, what kinds of AI-like functionality are included in the parts of the code that are actually verified? E.g. does the verified code include classical planning algorithms, modern planning algorithms, logical agent architectures, or perhaps even machine learning algorithms in some cases?

Diana: The code that gets verified consists of reactive, “anytime” plans, which are plans that get continually executed in response to internal and external environmental conditions. Each agent’s plan is a finite-state automaton (FSA), which consists of states and state-to-state transitions. Each state in the FSA corresponds to a subtask of the overall task (which is represented by the entire FSA). And each transition corresponds to an action taken by the agent. In general, there are multiple transitions exiting each state, corresponding to the choice of action taken by the agent. For example, consider the scenario I described in one of my previous answers in this interview, i.e., that of a planetary lander along with rovers. Two possible states for a planetary lander L might be “TRANSMITTING DATA” and “RECEIVING DATA.” Suppose the lander is in the former state. If it takes the action “PAUSE” then it will stay in its current state, but if it takes the action “TRANSMIT” then after this action has completed the lander will transition to the latter state.  Furthermore, the conditions for transitioning from one state to the next depend not only on what action the agent takes, but also on what’s going on in the environment, including what this agent observes the other agents (e.g., the rovers) doing. For this reason, I call the plans “reactive.”
Every FSA has an initial state, but no final state. The philosophy is that the agents are indefinitely reactive to environmental conditions subsequent to task initiation, and their task is continually ongoing. FSAs are internally represented as finite graphs, with vertices (nodes) corresponding to behavioral states and directed edges corresponding to state-to-state transitions.
Machine learning (ML) is applied to the FSA plans for the purposes of agent initialization and adaptation. Learning is done with evolutionary algorithms (EAs), using traditional generalization and specialization operators. These operators add, delete, move, or modify vertices and edges, as well as actions associated with the edges.  For example, suppose the lander’s transition from its “TRANSMITTING DATA” to its “RECEIVING DATA” state depends not only on its own “TRANSMIT” action, but it also requires that rover R1 successfully received the data transmitted by lander L before the lander can make this state-to-state transition in its FSA. This is a very reasonable requirement. Now suppose that R1’s communication apparatus has catastrophically failed. Then L will need to adapt its FSA by modifying the requirement of checking R1’s receipt of the transmission before it can transition to its “RECEIVING DATA” state. One possibility is that it replaces “R1” with “R2” in its FSA. Many other alternative learning operators are of course also possible, depending on the circumstances.
Machine learning is assumed to occur in two phases: offline then online. During the offline initialization phase, each agent starts with a randomly initialized population of candidate FSA plans, which is then evolved using evolutionary algorithms. The main loop of the EA consists of selecting parent plans from the population, applying ML operators to produce offspring, evaluating the fitness of the offspring, and then returning the offspring to the population if they are sufficiently “fit.” After evolving a good population of candidate plans, the agent then selects the “best” (according to its fitness criteria) plan from its population. Verification is then performed to this plan as well as repair, if required. During the online phase, the agents are fielded and plan execution is interleaved with learning (adaptation to environmental changes, such as agent hardware failures), re-verification, and plan repair as needed.
The main point of my “Asimovian adaptive agents” paper is that by knowing what adaptation was done by the agent, i.e., what machine learning operator was applied to the FSA, we can streamline the re-verification process enormously.

Luke: AI systems are becoming increasingly autonomous in operation: self-driving cars, robots that navigate disaster sites, HFT programs that trade stocks quickly enough to “flash crash” the market or nearly bankrupt a large equities trader, etc.
How might current AI safety methods (formal verification and reverification, program synthesis, simplex architectures, hybrid systems control, layered architectures, etc.) be extended to meet the safety challenges that will be raised by the future’s highly autonomous systems operating in unknown, continuous, dynamic environments? Do you think our capacity to make systems more autonomous and capable will outpace our capacity to achieve confident safety assurances for those systems?

Diana: My response to your first question is problem- and context-dependent. I know of many AI communities that are built around a single algorithm, and the researchers in those communities try to apply this algorithm to as many problems as possible. I believe that’s a misguided approach to research. Instead, I have always tried to adopt a problem-driven approach to research. The best way to scientifically solve a problem is to study it in great depth, and based on the a priori problem/task analysis select the most appropriate solution — including the planner or problem-solver, the properties/constraints to be verified, the adaptation method(s), etc. This will require a large suite of different AI safety/verification methods from which to choose. I cannot foresee the nature of this suite in advance; it’ll have to be constructed based on experience. As we tackle more complex autonomous systems, our repertoire of verification techniques will grow commensurately.
Your second question about whether autonomy will outpace safety is an extremely important one, Luke. Based on the applications you listed in your first paragraph, I see that your concerns extend to security. In fact, your security concerns also apply to “the internet of things,” which includes electronic, interconnected, remotely-accessible autonomous devices such as washing machines, ovens, and thermostats that will be installed in “smart homes” of the future. Businesses usually lack the motivation to install safety and security measures without some incentive. For example, leading software companies release beta versions of their programs with the expectation that the public will find and report the bugs. This is unacceptable as we transition to increasingly capable and potentially hazardous autonomous systems. I believe that the primary incentive will be government regulations. But we can’t wait until disasters arise before putting these regulations in place! Instead, we need to be proactive.
In 2008-2009 I was a member of a American Association for the Advancement of Artificial Intelligence (AAAI) Presidential Panel to study these issues. This was a fabulous panel, and it brought awareness to the AI community of researchers. Nevertheless it’s time raise awareness beyond the community of AI researchers. One suggestion I have is to assign a new or existing member of the United States President’s Council of Advisors on Science and Technology the task of studying and assessing the safety and security of autonomous systems.  This council member should consult the following people:

AI researchers who have extensive experience in developing autonomous systems
Engineers from aerospace, transportation, and other applications where safety is paramount
Lawyers and lawmakers who are cognizant of the legal issues that could arise
Cyber security experts.

I assume this council member would research the topic, consult others, conduct meetings, and conclude with a report and recommendations. Furthermore, I strongly believe that such a task should be assigned as soon as possible. We are already in a state where autonomy is outpacing safety and security, particularly in the commercial sector outside of the transportation industry.

Luke: Given that “autonomy is outpacing safety and security,” what are some other ideas you have for increasing the odds of reliably good outcomes from future autonomous systems?
By the way, I’ve only ever seen an “interim” report from that AAAI panel. Is there supposed to be a follow-up report at some point?

Diana: I haven’t heard about any follow-up or final report for the AAAI panel, unfortunately.
One idea is that we should have extensive safety and security testing prior to product release, based on established industry/government standards. We may not be able to enforce 100% compliance, but the presence of something like a “Safe and Secure Autonomous Product” certification could motivate consumers to favor purchasing tested and certified products over others lacking compliance. This would be like the existing UL product certification.
Another idea is to have monitoring, safety shutoff, self-recovery, and self-repair capabilities associated with autonomous devices. Furthermore, for security reasons these mechanisms should be decoupled from the autonomous system’s control, and they should also be detached from communications (e.g., not connected to the Internet) so as to avoid malicious tampering.
I don’t believe it’s possible to ensure complete safety and security at all times with autonomous systems. As you stated above, the best we can do is to increase “the odds of reliably good outcomes.” Nevertheless, I believe that substantial progress can be made if there is financial, technical, legal and programmatic support in this direction.

Luke: Thanks, Diana!
Perkins, T. and Barto, A. “Lyapunov design for safe reinforcement learning control.” Proceedings of AAAI’02. Margineantu, Schumann, Gupta, Drumheller, and Fresnedo (co-chairs). Workshop on “Verification, validation, and testing of learning systems.” NIPS’04. Dietterich, T. “Research issues in deployed adaptive systems.” NIPS’06.Schumann, J. “Applications of Neural Networks in High Assurance Systems.” Springer-Verlag, 2010.Stepanyan, V. et al., “Stability and performance metrics for adaptive flight control.” AIAA’09.Rushby, J. “A safety-case approach for certifying adaptive systems.” AIAA’09.Tiwari, A. “Bounded verification of adaptive flight control systems.” AIAA’10.Schurr, N. et al. “Asimovian multiagents: Laws of robotics to teams of humans and agents.” 2006.Schurr, N. “Toward human-multiagent teams.” USC Ph.D. dissertation, 2007.Levine, G. et al. “Learning and verifying safety constraints for planners in a knowledge-impoverished system.” Computational Intelligence 28 (3), 2012.Tamura, G. et al. “Towards practical runtime verification and validation of self-adaptive software systems.” Self-Adaptive Systems, LNCS 7475, Springer-Verlag, 2013.Zhang et al. “Modular verification of dynamically adaptive systems.” AOSD’09.Sharifloo, A. and Spoletini, P. “LOVER: Light-weight formal verification of adaptive systems at run time.” Formal Aspects of Component Software. Lecture Notes in Computer Science Volume 7684, pp. 170-187, 2013.Musliner , D. et al. “Incremental verification for on-the-fly controller synthesis.” MoChArt’05.Rubinfeld, R. Checking.Sokolsky, O. Selected Recent Publications by SubjectGordon, D. et. al. “Distributed spatial control, global monitoring and steering of mobile physical agents.” ICIIS’99.Spears, W. and Spears, D. (Eds.) “Physicomimetics: Physics-Based Swarm Intelligence.” Springer-Verlag, 2012.Spears, W. and Spears, D. (Eds.) 2012.DARPA-sponsored ISAT meeting on “Trustable Deployed Adaptive Systems” at SRI, 2006.The post Diana Spears on the safety of adaptive agents appeared first on Machine Intelligence Research Institute.


Will MacAskill on normative uncertainty

 Will MacAskill recently completed his DPhil at Oxford University and, as of October 2014 will be a Research Fellow at Emmanuel College, Cambridge.
He is the cofounder of Giving What We Can and 80,000 Hours. He’s currently writing a book, Effective Altruism, to be published by Gotham (Penguin USA) in summer 2015.

Luke Muehlhauser: In MacAskill (2014) you tackle the question of normative uncertainty:
Very often, we are unsure about what we ought to do… Sometimes, this uncertainty arises out of empirical uncertainty: we might not know to what extent non-human animals feel pain, or how much we are really able to improve the lives of distant strangers compared to our family members. But this uncertainty can also arise out of fundamental normative uncertainty: out of not knowing, for example, what moral weight the wellbeing of distant strangers has compared to the wellbeing of our family; or whether non-human animals are worthy of moral concern even given knowledge of all the facts about their biology and psychology.
…one might have expected philosophers to have devoted considerable research time to the question of how one ought to take one’s normative uncertainty into account in one’s decisions. But the issue has been largely neglected. This thesis attempts to begin to fill this gap.
In the first part of your thesis you argue that when the moral theories to which an agent assigns some credence are cardinally measurable (as opposed to ordinal-scale) and they are intertheoretically comparable, the agent should choose an action which “maximizes expected choice-worthiness” (MEC), which is akin to maximizing expected value across multiple uncertain theories of what is desirable.
I suspect that result will be intuitive to many, so let’s jump forward to where things get more interesting. You write:
Sometimes, [value] theories are merely ordinal, and, sometimes, even when theories are cardinal, choice-worthiness is not comparable between them. In either of these situations, MEC cannot be applied. In light of this problem, I propose that the correct metanormative theory is sensitive to the different sorts of information that different theories provide. In chapter 2, I consider how to take normative uncertainty into account in conditions where all theories provide merely ordinal choice-worthiness, and where choice-worthiness is noncomparable between theories, arguing in favour of the Borda Rule.
What is the Borda Rule, and why do you think it’s the best action rule under these conditions?

Will MacAskill: Re: “I suspect that result will be intuitive to many.” Maybe in your circles that’s true! Many, or even most, philosophers get off the boat way before this point. They say that there’s no sense of ‘ought’ according to which what one ought to do takes normative uncertainty into account. I’m glad that I don’t have to defend that for you, though, as I think it’s perfectly obvious that the ‘no ought’ position is silly.
As for the Borda Rule: the Borda Rule is a type of voting system, which works as follows. For each theory, an option’s Borda Score is equal to the number of options that rank lower in the theory’s choice-worthiness ordering than that option. An option’s Credence-Weighted Borda Score is equal to the sum, across all theories, of the decision-maker’s credence in the theory multiplied by the Borda Score of the option, on that theory.
So, for example, suppose I have 80% credence in Kantianism and 20% credence in Contractualism. (Suppose I’ve had some very misleading evidence….) Kantianism says that option A is the best option, then option B, then option C. Contractualism says that option C is the best option, then option B, then option A.
The Borda scores, on Kantianism, are:
A = 2
B = 1
C = 0
The Borda scores, on Contractualism, are:
A = 0
B = 1
C = 2
Each option’s Credence-Weighted Borda Score is:
A = 0.8*2 + 0.2*0 = 1.6
B = 0.8*1 + 0.2*1 = 1
C = 0.8*0 + 0.2*2 = 0.4
So, in this case, the Borda Rule would say that A is the most appropriate option, followed by B, and then C.
The reason we need to use some sort of voting system is because I’m considering, at this point, only ordinal theories: theories that tell you that it’s better to choose A over B (alt: that “A is more choice-worthy than B”), but won’t tell you how much more choice-worthy A is than B. So, in these conditions, we have to have a theory of how to take normative uncertainty into account that’s sensitive only to each theory’s choice-worthiness ordering (as well as the degree of credence in each theory), because the theories I’m considering don’t give you anything more than an ordering.
The key reason why I think the Borda Rule is better than any other voting system is that it satisfies a condition I call Updating Consistency. The idea is that increasing your credence in some particular theory T1 shouldn’t make the appropriateness ordering (that is, the ordering of options in terms of what-you-ought-to-do-under-normative-uncertainty) worse by the lights of T1.
This condition seems to me to be very plausible indeed. But, surprisingly, very few voting systems satisfy that property, and those others that do have other problems.

Luke: One problem for the Borda Rule is that it is, as you say, “extremely sensitive to how one individuates options” — a problem analogous to the problem of clone-dependence in voting theory. You tackle this problem by modifying the Borda Rule to include a measure over the set of all possible options. Could you explain how that works? Also, is this modification to the Borda Rule novel to your thesis?

Will: A measure is a way of giving sense to the size of a space. It allows us to say that some options represent a larger part of possibility space than others. This is an intuitive idea: ‘drinking tea tomorrow’ represents a larger portion of possibility space than ‘drinking tea with my left hand tomorrow at 3pm’. With the idea of a measure on board, we can rewrite our definition of the Borda Rule, as follows (I’ll ignore the possibility of equally choice-worthy options, as that makes the definition a little more complicated):
For each theory, an option’s Borda Score is equal to the sum total of the measure of the options that rank lower in the theory’s choice-worthiness ordering than that option. An option’s Credence-Weighted Borda Score is equal to the sum, across all theories, of the decision-maker’s credence in the theory multiplied by the Borda Score of the option, on that theory.
So, suppose that, according to some theory Ti, A>B. On the old definition of the Borda Rule, A gets a Borda Score of 1. But if option B is split into options B’ and B”, such that A>B’=B”, then A gets a Borda score of 2. The fact that A’s score has changed just because of how you’ve individuated options is problematic.
But let’s use the new definition, which incorporates a measure, and suppose that the measure of A is 0.5 and the measure of B is 0.5. If so, then, when the decision-maker faces options A and B, then A gets a Borda Score of 0.5, on Ti. But when option B is split into options B’ and B”, then the measure is split, too. Suppose that B’ and B” are equally large. If so, then B’ would have a measure of 0.25 and B” would have a measure of 0.25. So A’s Borda score, on Ti, would be 0.5, just as before.
This modification to the Borda Rule is novel, though the idea was given to me by Owen Cotton-Barratt, so I can’t take credit! I guess the reason it hasn’t been suggested in the voting theory literature is because it might seem obvious that every candidate gets the same measure. But perhaps you could think of the ‘space’ of possible political positions (which would be easy if it were really a left-right spectrum), and assign candidates a measure based on how much of this space they take up. That could possibly allow for the Borda Rule to avoid problems to do with clone-dependence. But I think that for actual voting systems, the Schulze method is better than the Borda Rule. It’s clone-independent even without a measure, and is much less vulnerable to strategic voting than the Borda Rule is.

Luke: What is the relevance of Arrow’s impossibility theorem to your suggested use of a modified Borda Rule for handling normative uncertainty?

Will: I suggest that, in conditions of ordinal theories, we should exploit the analogy with voting. But that analogy with voting means that we’ll run into a analogue of Arrow’s Impossibility Theorem: the result that no voting system can satisfy all of a number of highly desirable properties.
There are a few ways to formulate the impossibility result. The strongest, in my view, is to say that any voting system that satisfies other more essential properties must violate Contraction Consistency, where Contraction Consistency is defined as follows:
Let A be the option set, and M be the set of maximally appropriate options within A. Let S be a subset of A that contains all members of M. The condition is: A is a maximally appropriate option, given option set S, iff it is a member of M.
It’s a condition that you’ve got to be careful how to formulate. I don’t go into that in my thesis. But some violations of it are intuitively clearly irrational. E.g. imagine you’ve got the options of blueberry ice cream or strawberry ice cream. You currently prefer blueberry. But then you discover that the restaurant also serves chocolate ice cream, and so you switch your preference from blueberry to strawberry, even though your assessment of the relative values of blueberry and strawberry hasn’t changed. That seems irrational – e.g. it would suggest that you should spend resources trying to find out if you have available to you an option that you know you won’t take.
I think that contraction consistency is a problem for the Borda Rule. But it’s a problem that affects all voting system analogues. So it’s something that we’ve got to live with – it’s just unfortunate that we have (or ought to have) credence in merely ordinal theories.
There is a second response as well. Which is to distinguish the Narrow and Broad versions of the Borda Rule. The Narrow version assigns Borda Scores within an option-set. The Broad version assigns Borda Scores across all possible options. It’s only the Narrow version that violates Contraction Consistency. But the Broad version has its own weirdnesses. Suppose that you’ve got a situation:
99%: T1: A>B
1%: T2: B>A
Where T1 and T2 are merely ordinal theories. It might seem obvious that you should pick A in that situation. But you can’t infer that from that case, according to the Broad Borda Rule. Instead, you’ve got to look at how A and B rank in T1 and T2’s ordering of all possible options. If A and B are very close on T1 but very far apart, on T2, then B might be the most appropriate option. So the Broad Borda is very difficult to use. And it gives results that seem wrong to me – as if you’re ‘faking’ cardinality where there is none.
So my general view on this is that any account you have will have deep problems. Endorsing a particular view involves carefully weighing up different strengths and weaknesses; there’s no obviously correct position. (This becomes a theme when you start working on normative uncertainty. To an extent, this should be expected: we’re dealing with messy nonideal agents, who don’t have perfect access to their own values or to the normative truth).

Luke: Your thesis covers many other interesting topics; we won’t try to cover them all here. How would you summarize the other major “takeaways” you’d most want people to know about from your thesis?

Will: The “Maximise Expected Choice-Worthiness” approach to moral uncertainty is the best approach. It is able to respond to a number of objections that have been levelled against it.
If you think that you can’t compare choice-worthiness across theories, then you should normalise different theories at their variance. But I think that the arguments for intertheoretic incomparability don’t work. Instead, you should feel comfortable about using your intuitions about how different theories compare.
We can make sense of the idea of two theories T1 and T2 having exactly the value-ordering over options, but T1 thinking that everything is twice as important as T2 does. So ‘utilitarianism’ really refers to a class of theories, each of different levels of amplification.
Most of our intuitions about different Newcomb and related problems can be captured by maximising expected choice-worthiness over uncertainty about whether evidential or causal decision theory is true (with much higher credence in causal decision-theory than evidential decision theory). Taking decision-theoretic uncertainty into account puts causal decision theory on pretty strong grounds — you can respond to the intuitive and “Why Ain’cha Rich?” arguments in favour of evidential decision theory.
Moral philosophy provides a bargain in terms of gaining new information: doing just a bit of philosophical study or research can radically alter the value of one’s options. So individuals, philanthropists, and governments should all spend a lot more resources on researching and studying ethics than they currently do.
Even if you think that continued human survival is net bad, you should still work to prevent near-term human extinction, in case the human race gets evidence to the contrary over the next few centuries. (Well, this is true given a few fairly controversial premises.

Luke: Thanks, Will!
The post Will MacAskill on normative uncertainty appeared first on Machine Intelligence Research Institute.

Will MacAskill on normative uncertainty