Milind Tambe on game theory in security applications

Milind Tambe is Helen N. and Emmett H. Jones Professor in Engineering at the University of Southern California (USC). He is a fellow of AAAI (Association for Advancement of Artificial Intelligence) and ACM (Association for Computing Machinery), as well as recipient of the ACM/SIGART Autonomous Agents Research Award, Christopher Columbus Fellowship Foundation Homeland security award, the INFORMS Wagner prize for excellence in Operations Research Practice, the Rist Prize of the Military Operations Research Society, IBM Faculty Award, Okawa Foundation Faculty Research Award, RoboCup scientific challenge award, USC Associates Award for Creativity in Research and USC Viterbi School of Engineering use-inspired research award.
Prof. Tambe has contributed several foundational papers in agents and multiagent systems; this includes areas of multiagent teamwork, distributed constraint optimization (DCOP) and security games. For this research, he has received the "influential paper award" from the International Foundation for Agents and Multiagent Systems (IFAAMAS), as well as with his research group, best paper awards at a number of premier Artificial Intelligence Conferences and workshops; these have included multiple best paper awards at the International Conference on Autonomous Agents and Multiagent Systems and International Conference on Intelligent Virtual Agents.
In addition, the "security games" framework and algorithms pioneered by Prof. Tambe and his research group are now deployed for real-world use by several agencies including the US Coast Guard, the US Federal Air Marshals service, the Transportation Security Administration, LAX Police and the LA Sheriff's Department for security scheduling at a variety of US ports, airports and transportation infrastructure. This research has led to him and his students receiving the US Coast Guard Meritorious Team Commendation from the Commandant, US Coast Guard First District's Operational Excellence Award, Certificate of Appreciation from the US Federal Air Marshals Service and special commendation given by the Los Angeles World Airports police from the city of Los Angeles. For his teaching and service, Prof. Tambe has received the USC Steven B. Sample Teaching and Mentoring award and the ACM recognition of service award. Recently, he co-founded ARMORWAY, a company focused on risk mitigation and security resource optimization, where he serves on the board of directors. Prof. Tambe received his Ph.D. from the School of Computer Science at Carnegie Mellon University.

Luke Muehlhauser: In Tambe et al. (2013), you and your co-authors give an overview of game theory in security applications, saying:
Game theory is well-suited to adversarial reasoning for security resource allocation and scheduling problems. Casting the problem as a Bayesian Stackelberg game, we have developed new algorithms for efficiently solving such games to provide randomized patrolling or inspection strategies.
You then give many examples of game-theoretic algorithms used for security at airports, borders, etc.
Is there evidence to suggest that the introduction of these systems has improved the security of the airports, borders, etc. relative to whatever security processes they were using before?

Milind Tambe: This is an important and wonderful question. There is a long answer to this question that compiles all of our evidence (this is actually a book chapter). I will attempt to provide a shorter answer here and will attempt to provide pointers to our publications that compile this evidence.
As you note, we are fortunate that many of our game-theoretic algorithms for security resource optimization (via optimal scheduling, allocation) have jumped out of our lab and are now deployed for real use in many applications. As we write papers about these deployed applications, evidence showing the benefits of these algorithms is definitely an important issue that is necessary for us to answer. Unlike our more "mainstream" papers, where we can run 1000s of careful simulations under controlled conditions, we cannot conduct such experiments in the real world with our deployed applications. Nor can we provide a proof of 100% security – there is no such thing. So what can we do? In our evidence gathering, we have focused on the question of: are we better off with our tools based on computational game theory than what was being done previously, which was typically relying on human schedulers or a simple dice roll for security scheduling (simple dice roll is often the other "automation" that is used or offered as an alternative to our methods). Now within my area of Artificial Intelligence, that AI programs can beat humans at complex scheduling tasks is not controversial; but regardless we have used the following methods to provide evidence that our game-theoretic algorithms indeed perform better. These methods range from simulations to actual field tests.

Simulations (including using a "machine learning" attacker): We provide simulations of security schedules, e.g., randomized patrols, assignments, comparing our approach to earlier approaches based on techniques used by human schedulers. We have a machine learning based attacker who learns any patterns and then chooses to attack the facility being protected. Game-theoretic schedulers are seen to perform significantly better in providing higher levels of protections (Pita et al. 2008; Jain et al. 2010).
Human adversaries in the lab: We have worked with a large number of human subjects and security experts (security officials) to have them get through randomized security schedules, where some are schedules generated by our algorithms, and some are baseline approaches for comparison. Human subjects are paid money based on the reward they collect by successfully intruding through our security schedules; again our game-theoretic schedulers perform significantly better (Pita et al. 2009).
Actual security schedules before and after: For some security applications, we have data on how scheduling was done by humans (before our algorithms were deployed) and how schedules are generated after deployment of our algorithms. For measures of interest to security agencies, e.g., predictability in schedules, we can compare the actual human-generated schedules vs our algorithmic schedules. Again, game-theoretic schedulers are seen to perform significantly better by avoiding predictability and yet ensuring that more important targets are covered with higher frequency of patrols. Some of this data is published (Shieh et al. 2012).
"Adversary" teams simulate attack: In some cases, security agencies have deployed adversary perspective teams or "mock attacker teams" that will attempt to conduct surveillance to plan attacks; this is done before and after our algorithms have been deployed to check which security deployments worked better. This was done by the US Coast Guard indicating that the game-theoretic scheduler provided higher levels of deterrence (Shieh et al. 2012).
Real-time comparison: human vs algorithm: This is a test we ran on the metro trains in Los Angeles. For a day of patrol scheduling, we provided head-to-head comparison of human schedulers trying to schedule 90 officers on patrols vs an automated game-theoretic scheduler. External evaluators then provided an evaluation of these patrols; the evaluators did not know who had generated each of the schedules. The results show that while human schedulers required significant effort even for generating one schedule (almost a day), and the game-theoretic scheduler ran quickly, the external evaluators rated the game theoretic schedulers higher (with statistical significance). (Delle Fave et al. 2014a, Delle Fave et al. under consideration).
Actual data from deployment: This is another test run on the metro trains in LA. We had a comparison of game-theoretic scheduler vs an alternative (in this case a uniform random scheduler augmented with real time human intelligence) to check fare evaders. In 21 days of patrols, the game-theoretic scheduler led to significantly higher numbers of fare evaders captured than the alternative. (Delle Fave et al. 2014a, Delle Fave et al. 2014b).
Domain expert evaluation (internal and external): There have been of course significant numbers of evaluations done by domain experts comparing their own scheduling method with game theoretic schedulers and repeatedly the game theoretic schedulers have come out ahead. The fact that our software is now in use for several years at several different important airports, ports, air-traffic, and so on, is an indicator to us that the domain experts must consider this software of some value.

All of this evidence suggests that the game-theoretic approaches for security resource optimization performs significantly better than its competitors, i.e., human schedulers or simple randomization. Humans were seen to fall into predictable patterns; and indeed this is an extremely complex adversarial scheduling/planning/allocation task, where humans have to reason about trillions of possible schedules (actually even more), and which is time-consuming and very difficult for humans. It would seem then that we should let this complex task to be handed over to software, and let humans concentrate on the more important tasks of actually physically providing security.

Luke: Diana Spears mentioned that you teach a course on "Artificial Intelligence and Science Fiction," which includes a section on your "research on Asimovian multiagents." Is this 2007 syllabus still a pretty accurate outline of the course? What work on "Asimovian multiagents" do you discuss? She mentioned Schurr et al. (2006); is there any other work on that topic you've done?

Milind: The last iteration of "Understanding intelligent agents via science fiction" was taught in 2010. Here is the syllabus for it.
This course was jointly developed with my former PhD student, Prof. Emma Bowring, now Associate Professor at Univ of Pacific, and really was her idea to develop it while she was still a student at USC. We have written about this course (Bowring & Tambe 2009).
While we did use Asimov's short stories extensively, the goal here is to really introduce core concepts in agents and multiagent systems, as can be seen from the syllabus. So the focus is more on using these stories, Star Trek clips, and other science fiction as a motivation to introduce core concepts in AI/Agents and Multiagent systems, starting from Markov Decision Problems (MDPs), POMDPs, Game theory, agent modeling and so on.
While you are right that we have done some follow up work based on Diana's original paper and earlier paper by Weld & Etzioni (1994) and that is a fascinating thread of research, but that isn't the focus of this course. This is more of an undergraduate course introducing students to key concepts. Towards the end of the course we get into abstract ideas on agent design where students may use some of the ideas advocated in these "Asimovian agents", which is great.
We haven't pursued that research direction beyond that paper and an earlier one (Pynadath and Tambe 2001).
It would be a great idea to push that direction more though.

Luke: In Tambe et al. (2013) you identify open research issues for game theory in security applications, such as scalability and robustness. Which open research problems in this area do you think would have the largest practical impact if they could be solved well?

Milind: Both scalability and robustness are important. Scale-up is important because we want to solve large-scale games. For example, even if we leave aside complex scheduling constraints and just think of the abstract assignment problem of security agencies such as Federal Air Marshals Service of assigning say 20 defenders to 1000 flights, that is 1000-choose-20. These problems become so large that we cant fit games in the normal form in memory and must somehow find an optimal defender strategy without explicitly representing the game in memory. On the other hand, we wish to handle robustness because of the many uncertainties in the game. There is uncertainty related to adversary's surveillance (how much surveillance is actually going on), uncertainty about adversary's payoffs, capabilities, and so on. When we combine the two requirements of solving large-scale games and handling uncertainty, then the problem becomes even more complex. These remain critical challenges for us to address.
There are however many other major research challenges that are open. Significant effort has been focused on modeling adversary bounded rationality. This exciting research at the intersection of algorithmic and behavioral game theory is a major area of research in security games. Furthermore, our recent work focused on protecting wildlife and fisheries (Yang et al. 2014; Haskell et al. 2014) brings up challenges related to Machine Learning. Specifically we now have data related to say poachers' movements and actions. This data can be used to create a better model of the poachers. Another area is preference elicitation. If there is significant uncertainty related to many aspects of the domain, and reducing this uncertainty is going to take effort, then which features do we focus on first to reduce uncertainty? Our recent paper at AAAI'2014 (Nguyen et al. 2014) provides our initial thrust into this area.
In short, while we have made significant progress, not just in my group, but as the "security games community," there is a lot more that still needs to be done.

Luke: Thanks, Milind!
The post Milind Tambe on game theory in security applications appeared first on Machine Intelligence Research Institute.


New report: "Loudness: On priors over preference relations"

Today we release the first technical report from our May 2014 workshop: "Loudness: on priors over preference relations" by Benja Fallenstein and Nisan Stiennon. Other technical reports from that workshop are also in progress. Here's the abstract for this report:
This is a quick writeup of a problem discussed at the May 2014 MIRI workshop: how to formally deal with uncertainty about preferences. We assume that the true preferences satisfy the von Neumann-Morgenstern (VNM) axioms, and can therefore be represented by a utility function. It may seem that we should then simply maximize the expectation of this function. However, in the absence of more information, this is not well-defined; in this setting, different choices of utility functions representing the same VNM preferences can lead the agent to make different choices. We give a formalization of this problem and show that the choice of a prior probability distribution over VNM preference relations together with the choice of a representative for each of these distributions is in a certain sense equivalent to the choice of a single number for every preference relation, which we call its "loudness". (Mathematically, a "loudness prior" can be seen as a probability distribution over preference relations, but this object does not have an epistemic interpretation.)
The post New report: "Loudness: On priors over preference relations" appeared first on Machine Intelligence Research Institute.

New report: “Loudness: On priors over preference relations”


MIRI wants to fund your independently-organized Friendly AI workshop

To support Friendly AI research around the world, our new MIRIx program funds mathematicians, computer scientists, and formal philosophers to organize their own Friendly AI workshops.
A MIRIx workshop can be as simple as gathering some of your friends to read MIRI papers together, talk about them, eat some snacks, scribble some ideas on whiteboards, and go out to dinner together. Or it can be a multi-day research workshop pursuing a specific line of attack on a particular problem. It's up to you.
Apply for funding here. In some cases we'll be able to send a MIRI research fellow to your first meeting to give tutorials and answer questions, or perhaps they'll Skype in to your workshop to do the same.
The post MIRI wants to fund your independently-organized Friendly AI workshop appeared first on Machine Intelligence Research Institute.


Aaron Tomb on crowd-sourced formal verification

 Aaron Tomb is a Principal Investigator at Galois, where his work includes research, development, and project leadership in the area of automated and semi-automated techniques for analysis of software, including type systems, defect detection tools, formal verification, and more general software development tools based on deep analysis of program semantics. He joined Galois in 2007, and received a Ph.D. in Computer Science from the University of California, Santa Cruz on a technique for identifying inconsistent assumptions in software source code.

Luke Muehlhauser: DARPA's Crowd Sourced Formal Verification (CSFV) program "aims to investigate whether large numbers of non-experts can perform formal verification faster and more cost-effectively than conventional processes." To that end, the program created Verigames.com, the home of several free online puzzle games.
Your contribution to Verigames is Stormbound. In a blog post, your colleague Jef Bell explains:
Here's a high level view of how StormBound works, from the [formal verification] perspective. First we figure out the properties of the system we want to prove…
Based on our knowledge about the common sources of errors we are trying to prevent, we select certain code points in the program source code where we need to check that a property we care about is in fact true…
Unfortunately, in a typical program there are a lot of these code points, and it is not always easy to find the exact property that needs to hold at each point. It turns out this is where StormBound players can help, so bear with me as I explain how…
We run the system a whole bunch of times in a variety of scenarios that are intended to exercise the system as it would be exercised in real-world use. As we run the system all these times, we collect snapshots of the data that is in the computer's memory, every time the program gets to one of the code points we identified.
This provides us with a large collection of examples of the data in the computer's memory. Now, we want to identify some things in common about the data at each code point, every time that same code point is reached while running the program. These things in common are in fact the kinds of properties we are looking for. And here's an important bit: it turns out that computers aren't always good at finding these "things in common," but human beings are very adept at finding patterns, especially visual patterns.
This is where the StormBound game comes in. Each game level actually consists of the data that was in memory all the times a particular code point was reached. When you play a level, you are looking for common patterns in that data. The game shows you the data (in what we think is a fun and visually appealing form), and allows you to "experiment" with the data in various ways, to help you to find those common patterns. The spells you cast in StormBound describe the patterns. In the [formal verification] world we call these spells assertions. We combine together the assertions discovered by many players of the same game level, which enables the [formal verification] tools we use to prove the truthfulness of the desired properties at the corresponding code point.
Roughly how many players have played StormBound so far? Have they yet found anything that you expect would have been very difficult for an automated search algorithm to find?

Aaron Tomb: At this point we've had thousands of players, who have collectively described over 100,000 patterns (which ultimately become logical assertions about program states). The game that is available for play at the moment is the first of two phases, and allows players to describe a relatively restricted class of assertions. And, in general, the assertions we have gathered during this first phase have been relatively simple, and likely discoverable using automated techniques. However, having the current iteration of the game available to a wide range of players has taught us a lot about what keeps players engaged (or causes them to lose interest), and has also allowed us to develop an understanding of the sorts of results that we need to get from players but that aren't encouraged (or, in a few cases, possible) with the current interface. We are using this information to design a second version that we expect will yield richer results beyond what automated tools would be likely to produce.

Luke: Makes sense. That's standard Silicon Valley wisdom: launch a minimum viable product, learn, iterate quickly, etc.
How would you know whether Stormbound, or even CSFV altogether (if you were running it), had succeeded or failed? Presumably there's a substantial chance that crowd-sourced formal verification assistance can't really work with anything like current verification techniques and problems?

Aaron: I can't speak for DARPA about how they plan to measure the success of the program, but I can say something about how we can know when we've successfully verified a particular piece of software. One advantage of typical verification techniques is that there's a fairly clear measure of completion. Verification tools often accept program source code plus some statement of desired properties of the code. The tools will then attempt to prove, usually automatically, that the code does indeed have those properties. If they succeed, there's a high probability (not 100%, because the tools may have bugs, themselves) that the code has the desired properties.
StormBound is structured to generate annotations for C code that are sent to a sophisticated tool called Frama-C. Those annotations are combined with some built-in definitions that describe the absence of common security vulnerabilities. By using Frama-C, StormBound is already working with state-of-the-art verification techniques. Our target programs are large pieces of open source software critical to internet infrastructure, so we're also working with real-world problems.
A key question is then: can StormBound generate the necessary annotations to allow verification to succeed? In the first phase, we've found that players have come up with some useful annotations, but have missed others that would be necessary for a complete proof. In the second phase, we'll be updating the game to include significantly more context about the ultimate verification goal, in game-oriented terms, and we expect that this will lead more players toward constructing solutions that give significant verification progress.

Luke: You write that "Our target programs are large pieces of open source software critical to internet infrastructure…"
Do you have any opinion as to whether formal verification would have been likely to catch the Heartbleed bug in OpenSSL?

Aaron: Yes, any sound verification tool capable of proving the absence of undefined behavior in C, such as Frama-C, would have failed to prove that OpenSSL was safe. Depending on the tool being used, that proof failure may or may not help you identify the nature of the bug, but it would at least indicate that some bug of some sort exists, and something about its rough location. John Regehr, a Computer Science professor at the University of Utah, wrote a nice blog post on the topic, which specifically mentions using Frama-C to do the verification.
The key complication is that, in order to prove that such bugs don't exist, tools like Frama-C require a lot of extra information from the programmer (or from game players, in the case of StormBound), encoded as annotations in the source code. In general, the effort required to create those annotations can exceed the time taken to write the program in the first place, and the process requires familiarity with verification tools, which is not widespread. We hope that StormBound will be able to significantly reduce the necessary effort and expertise, to the point where it becomes practical.

Luke: What's your current guess as to how general the CSFV of 5-10 years from now will be? How broad a range of verification challenges might they conceivably be applied to, once they are substantially more fully developed than they are in these early days?

Aaron: A lot can happen in 5-10 years, so my answer to this is very speculative, and just reflects my own opinions, not those of DARPA or even the rest of the StormBound team.
In the last decade, there has been significant progress on tools for interactive construction of mathematical proofs that can be reliably checked by computers for correctness. Some very general and robust tools such as ACL2, Coq, Isabelle and PVS have arisen from this research. (Most of them were initially created even earlier than the last decade, but the approach has become significantly more widespread recently.) These tools have been used for significant undertakings in software verification, as well as general mathematics, resulting in successful proofs that would simply be infeasible to do (correctly) by hand. However, these tools are time-consuming to use for proofs about realistic software systems, and require rare expertise.
The design approaches we have been considering for next revision of StormBound borrow significantly from the technical underpinnings of these interactive theorem provers, and the resulting game should, as a result, be quite general. While the tentative design builds on these ideas, it presents them through a very different interface than that used by existing tools, and we hope that will allow people without the same level of mathematical background to prove at least some theorems.
The key unknown, at the moment, is how well players without mathematical background will be able to develop the intuition necessary to make progress. The current design sketches focus heavily on using visual pattern recognition in place of standard mathematical reasoning techniques in many places, but proofs of typical mathematical theorems still tend to rely on deep insight based on years of experience. At the same time, the proofs necessary to verify some basic security properties of software are often much less intricate than those that mathematicians spend their time on.
So, to finally get to the answer, I could see several possible results the next decade. We could have a significantly different interface to interactive theorem provers that may make expert users more efficient. An even more exciting possibility, though, is that we could have a system that allows non-experts to take a stab at proving essentially any mathematical theorem simply by recognizing patterns and applying a few relatively simple rules. In either case, the I expect that the results will be more general than simply proving security properties about software.

Luke: Thanks, Aaron!
The post Aaron Tomb on crowd-sourced formal verification appeared first on Machine Intelligence Research Institute.


Lennart Beringer on the Verified Software Toolchain

 Lennart Beringer is an Associate Research Scholar at Princeton University, where he uses interactive proof assistants to develop provably correct compilers, program analyses, and other software verification tools. Previously, he held research appointments at the Ludwig-Maximilians-University Munich and the University of Edinburgh, where he developed proof-carrying-code techniques for mobile code architectures, focusing on properties of resource consumption and information flow security. He received his PhD degree from the University of Edinburgh for a language-based analysis of an idealized asynchronous processor architecture. In 2012, he served as the co-chair of the Third International Conference on Interactive Theorem Proving (ITP).

Luke Muehlhauser: One of the projects to which you contribute is the Verified Software Toolchain. How would you summarize what has been achieved so far? What do you hope to have achieved in the next year or two on that project?

Lennart Beringer: The Verified Software Toolchain is a collection of verification tools for the C programming language, implemented and formally verified in the Coq proof assistant, with a connection to the formally verified CompCert compiler developed by Xavier Leroy at INRIA, France.
The heart of the VST is a Hoare-style program logic suitable for verifying partial-correctness properties and program safety, i.e. the absence of abnormal program termination due to, e.g., an attempt to dereference a dangling pointer. Program specifications are written in a highly expressive variant of concurrent separation logic that supports higher-order reasoning, impredicative quantification, and other concepts necessary for formulating, e.g., the resource invariants of locks in POSIX-style multithreading.
In addition to the core program logic, the VST contains support for proof automation, and a verifiably sound implementation of a shape analysis, the two components of which — a symbolic execution engine and an entailment checker for the Smallfoot-fragment of separation logic — can be either code-extracted to yield stand-alone tools, or integrated into Coq- or Ocaml-based analyses, including Smallfoot itself. The proof automation component consists of various Coq tactics, auxiliary lemmas, and specification abstractions that capture typical proof patterns in concrete application programs.  Special emphasis here is put on the idea of hierarchically organized layers of assertion formats that hide much of the complexity of the underlying logic from the user. An important goal of our current research is the discovery of additional such reasoning patterns for various programming and specification styles, and the exploitation of such structure to further enhance automation and performance. Regarding applications of the VST, we believe that in addition to the typical domains of high-assurance software, concrete open-source implementations of formal standards or libraries provide excellent case studies for evaluating further enhancements to the VST. At the same time, machine-checkable proofs (or even just precise specifications) of such libraries can contribute to the public level of trust that everyday users put on these libraries.
Many design decisions and implementation details of the VST are described in Andrew Appel's recent book, Program Logics for Certified Compilers (CUP 2014), while the source code is publicly available here.
A further aspect of our work concerns CompCert itself. Here, we are collaborating with Leroy and others to evolve CompCert's specification and proof to cover shared memory interaction between threads, and formal models of linking. Indeed, a modular notion of Compiler correctness should respect that communication with the OS, with libraries, or with separately compiled program modules in C crucially relies on exchanging pointers to buffers or other shared data structures. Instead, CompCert's notion of compiler correctness employs a whole-program view of compilation. However, it is far from easy to reconcile pointer sharing with the requirements of typical compiler optimizations, as these routinely alter memory layout and eliminate or relocate existing pointers. In addition, compiler-managed memory locations such as return addresses and spilling locations must remain unmodified be even those external callees that have legitimate access to other stack-held data.

Luke: How would you describe the underlying motivation for the VST? How will the world be different for various kinds of programmers (and other actors) in 10 years if the VST continues to be funded and worked on at a good pace?

Lennart: Specifying the expected behavior of software, and verifying that concrete implementations satisfy these specifications, has been an active research area for several decades. But for many years, the effectiveness of program logics was limited, and one of the reasons for this was the lack of support for modular reasoning, particularly with respect to pointer sharing and data structures that are laid out in memory. About ten years ago, separation logics emerged as a promising technique to address this shortcoming, by allowing specifications and invariants to elegantly capture properties of the heap and other shared resources.
At the same time, interactive proof assistants (and the computation power of the underlying hardware) had sufficiently matured so that program logics could be effectively implemented, and could be equipped with machine-checkable proofs of soundness. Indeed, by developing CompCert, Leroy demonstrated that interactive theorem proving scales to real-world languages, and can cope with the complexity inherent in an industrial-strength optimizing compiler.
The VST project combines these two threads, while at the same time exploring how the current automation and specification boundaries can be pushed further.
The main tool chain of the VST will most likely be of interest to C programmers writing either high assurance software, to implementors of medium-scale software such as crypto libraries, or to developers using code synthesis techniques, where — instead of verifying the correctness of individually generated programs — one could envision verified synthesis tools that target the VST's Clight program logic. Functional programmers may benefit by having a precise way to formally relate their high-level code written in Ocaml or Haskell to potentially more efficient code written in C, using Coq's code extraction feature to mediate between these two worlds. By combining code extraction and compilation, it may even become possible to verify hybrid software systems in which some components are written in C and others in a functional language.
Another group of potential users is that of system designers: VST specifications enable individual components to be characterized by much richer interfaces than is possible using C header files or Java interfaces. As interfaces become more robust, truly component-oriented software development becomes possible, where adherence to specifications suffices to guarantee that substituting individual components does not have negative impact on global system properties.
For typical components of the standard system stack (OS, network protocols,…), such formally defined interfaces are in fact of interest in their own right, as the contract between, say, different components of the OS, the compiler, and the architecture, are notoriously difficult to comprehend. In addition to system developers and researchers, a third clientele may hence be that of educators, for whom specifications offer guidance for partitioning their material into self-contained, but interdependent units of teaching, and for explaining the mutual contracts and assumptions.

Luke: What are some other projects that seem to be aimed at similar goals but e.g. for other languages or via other approaches?

Lennart: I am not aware of other attempts to develop a similarly expressive program logic for a mainstream language, validate its soundness in a proof assistant, link it with a verified optimizing compiler, and equip it with sufficient proof automation to enable the verification of nontrivial programs.
However, many research groups address various subsets of these features.
Microsoft Research over the past decade developed an impressive collection of verification tools (VCC, Boogie/Dafny,…) which are primarily targeted at C#, but contain components or intermediate verification platforms that are also applicable to other programming languages. These have excellent proof automation, and are tightly integrated with modern SAT/SMT solvers, software model checkers, and other code analysis and inspection tools, but are not formally backed up by a machine-checkable soundness proof. At the same time, several groups within MSR already employ proof assistants to verify compilers, develop formal models of machine code, and validate program logics that are roughly comparable to the VST. It would hardly come as a surprise if these complementary research efforts were eventually integrated to yield an end-to-end machine-checkable toolchain for C#.
Chlipala's Bedrock framework enables synthesis of verified assembly code by providing machine-checked code and specification patterns, enabling interaction between code fragments that deviates from C's calling convention. In addition, this group pioneered proof engineering techniques that exploit advanced features of Coq's tactic language and type theory, including some approaches we are now applying in the VST. We also incorporated ideas from Jesper Bengtson's Charge! project, which provides a Coq-validated separation logic for Java.
Regarding compiler verification, Xavier Leroy and his collaborators at INRIA are extending CompCert towards full support for both C99 and C11, at the same time validating further optimization phases and – most recently – adding a verified parser.
UPenn's Vellum project developed mechanized reasoning support for programs expressed in the LLVM intermediate representation, demonstrating that CompCert's verification principles are also applicable to compiler infrastructures that were not designed with formal verification in mind, and to compilation based on static single assignment (SSA) in particular.
Closely related to compiler verification is the approach of translation validation – in fact, CompCert's register allocator is also proved in this style. Recent contributions to mechanized translation validation include those by Sorin Lerner's group at San Diego, and Greg Morrisett's group at Harvard.
The development of practically validated, yet formally precise, mechanized specifications of real-world hard- and software systems, is an area that continues to grow, with recent examples including web servers, browsers, databases, operating system and hypervisors, relaxed memory models, and processor architectures. Particularly noteworthy are the projects under DARPA's CRASH programme, as well as the REMS effort led by Peter Sewell at Cambridge (UK).
I probably failed to mention a number of projects, but I hope that the ones I mentioned give you a rough picture of the current developments.

Luke: Thanks, Lennart!
The post Lennart Beringer on the Verified Software Toolchain appeared first on Machine Intelligence Research Institute.


Johann Schumann on high-assurance systems

Dr. Johann Schumann is a member of the Robust Software Engineering Group RSE at NASA Ames. He obtained his habilitation degree (2000) from the Technische Universität München, Germany on application of automated theorem provers in Software Engineering. His PhD thesis (1991) was on high-performance parallel theorem provers. Dr. Schumann is engaged in research on software and system health management, verification and validation of advanced air traffic control algorithms and adaptive systems, statistical data analysis of air traffic control systems and UAS incident data, and the generation of reliable code for data analysis and state estimation. Dr. Schumann's general research interests focus on the application of formal and statistical methods to improve design and reliability of advanced safety- and security-critical software.
He is employed by SGT, Inc. as Chief Scientist Computational Sciences.

Luke Muehlhauser: In Denney et al. (2006), you and your co-authors helpfully explain:
Software certification aims to show that the software in question achieves a certain level of quality, safety, or security. Its result is a certificate, i.e., independently checkable evidence of the properties claimed. Certification approaches vary widely… but the highest degree of confidence is achieved with approaches that are based on formal methods and use logic and theorem proving to construct the certificates.
We have developed a certification approach which… [demonstrates] the safety of aerospace software which has been automatically generated from high-level specifications. Our core idea is to extend the code generator so that it simultaneously generates code and the detailed annotations… that enable a fully automated safety proof. A verification condition generator (VCG) processes the annotated code and produces a set of safety obligations, which are provable if and only if the code is safe. An automated theorem prover (ATP) then discharges these obligations and the proofs, which can be verified by an independent proof checker, serve as certificates…
…[Our] architecture distinguishes between trusted and untrusted components… Trusted components must be correct because any errors in their results can compromise the assurance given by the system; untrusted components do not affect soundness because their results can be checked by trusted components.
You also include this helpful illustration:

Why are the problem specification and synthesis system untrusted, in your certification system architecture?

Johann Schumann: Ideally, the entire architecture in this figure would only contain trusted components. Then one could be sure that nothing bad happens during the verification process. Unfortunately, software systems like the program synthesis system or an entire automated theorem prover cannot be formally verified with current technology. Their code is tens of thousands of lines of code. We therefore use a trick of problem complexity asymmetry: E.g., a piece of software to find the prime factors of a large number is complex; its verification is complicated and time consuming. On the other hand, checking each individual result is trivial: one just multiplies the factors. Such a checker is easy to implement and to verify. So, if we now use the unverified prime-factor-finder and, for each result, run the quick and small verified checker, we can find out if our prime-factor finder works correctly. Important here is: one needs to run the checker for each result, but since the checker is fast, it doesn't matter. Similar situation here: synthesis system and prover are large and complicated, but the proof checker isn't. The VCG is of medium complexity and can be verified. The practical bottleneck of this entire system is that you have to trust your safety policy and the domain theory. Both can be larger set of logic formulas. In my experience: if a formula is longer than 5 lines, it is very likely that it contains a bug. Unfortunately, there is no easy way around that.

Luke: Intuitively, how much "safer" do you think a system developed via formal methods (such as your own process) is, compared to a system developed via normal engineering methods and extensive testing? Have there been any empirical tests of such hypotheses, to your knowledge?

Johann: Intuitively, I think that the use of formal methods (FM) for software development and V&V can substantially increase safety. However, there are a number of issues/considerations:

Using FMs to verify that certain, well understood safety properties are never violated (e.g., no buffer overflow, no divide-by-zero, no uninitialized variables) is extremely beneficial (see recent security problems — a lot of those have to do with such safety properties). Some technologies for that task (e.g., static analysis) have already achieved much in that direction.
Even assuming the tools are there, one has to ask: can i specify all necessary security properties? Is the specification of the software, against which I do the FMs based verification, correct and consistent? A correct, consistent, and complete specification is extremely hard to write — so for any software of reasonable complexity, I would assume that there are at least as many bugs in the specification as there are in the code.
In my opinion, FMs should be used together with traditional methods (including testing) and together with dynamic monitoring. As an example, a formal proof of no-buffer-overflow should be able to reduce (but not eliminate) extensive testing. And since testing only can be finite, dynamic monitoring (aka runtime verification (RV) or software health management (SWHM)) should be used to react on unforeseen software behavior (e.g., caused by dormant bugs or operation in an unexpected environment). RV and SWHM themselves are mostly formal-methods based.

I don't know of any such empirical tests – they would be very hard to carry out in an unbiased way. Work on cost and reliability modeling (e.g., B. Boehm) might give some insight but I would need to look.

Luke: How effectively do you think formal verification methods will eventually be able to handle more complex safety properties, such as collision avoidance for trains, planes and cars, and Asimovian properties (a la Weld & Etzioni) for systems that interact with humans in general?

Johann: There have been a couple of examples, where formal methods have been successfully used for specification and/or development of a complex system. For example, the B method (a tool-supported method related to the Z method) has been used for the automated train control on the Paris Metro (Line 14). Airbus is using formal methods like abstract interpretation (Cousot) for their development of safety-critical aircraft software.
The TCAS system (Traffic Collision Avoidance System) is a international system on commercial aircraft that warns the pilot from danger of an imminent midair collision (less than a minute away). This complex system (the aircraft need to resolve the conflict among themselves using an elaborate communication protocol) has been studied in detail and formal specifications/analyses exist (e.g., Leveson, Heimdahl and others).
In my opinion, formal methods are strong enough to be used to specify/verify complex systems. However, the Ueberlingen accident showed that this system is not fail-safe. Here "system" means: actual hardware/software, the pilots (who got conflicting messages from the TCAS system and the ATC), the airline (which supposedly told the pilots to ignore warnings and fly the most profitable route), and the swiss air traffic control, which was not fully staffed.
To me it seems that the accurate modeling and prediction of human behavior in a mixed control system (where the human operator has all/partial control) is still a big unsolved issue.
A fully automated system (e.g., UAS) might have safely handled this situation, but in general there are several major issues:

The acceptance problem (would we board a UAS, would we relinguish the wheel of our cars?). Supposedly when BART first introduced driverless trains, they weren't accepted by the commuters and they had to put drivers back on the trains.
How to operate safely in a "mixed-mode" environment? E.g., some cars are automatic, some cars are driven manually, or commerical aircraft versus general aviation and UAS systems. Human operators might make a (silly) mistake or react in an unexpected way. The automated system might react unexpectedly too, e.g., because of a software error.
How to deal with system security? Many systems to increase safety are currently highly vulnerable to malicious attack: E.g., the ADS-B system, which is supposed to replace parts of the aging ATC system uses an unencrypted protocol; UAS aircraft can be spoofed (see CIA UAS captured by Iran); modern cars can be hacked into easily.

Although a lot has happened in the rigorous treament and specification of complex systems, a number of issues needs to be addressed before we even come close to talk about global properties like the laws of robotics.
How to recover a complex system safely after an operator error? Assume a driver in a car with driving assistence systems (e.g., adaptive cruise control, detection of unsafe lane changes). How does the automated system react to the imminent problem without causing the driver to freak out and make more mistakes?
How to recover the system safely after a problem in the system hardware or software? When the engine exploded on the Qantas A380, the pilots received on the order of 400 diagnostic and error messages, which they had to process (luckily they had the time). Some of the messages contradicted each other or would have been dangerous to carry out.
This shows that it is still a big problem to present the operator with concise and correct status information of a complex system.
After the pilots landed the A380, they couldn't turn of the engines. First sign of machine intelligence "don't kill me"?

Luke: Thanks, Johann!
The post Johann Schumann on high-assurance systems appeared first on Machine Intelligence Research Institute.


Sandor Veres on autonomous agents

 Professor Sandor Veres was born and educated in Hungary as applied mathematician. He completed his PhD in dynamical modelling of stochastic systems in 1983 and worked in industry on computer controlled systems. In <PHONE> he received two consecutive scholarships at Imperial College London and at Linacre College Oxford.
Between <PHONE> he was lecturer at the Electronic and Electrical engineering department at the University of Birmingham and pursued research in system identification, adaptive control and embedded electronic control systems. In 2000 he joined the University of Southampton to do work in the areas of active vibration control systems, adaptive and learning systems, satellite formation flying and autonomous control. Since 2002 he has held a chair in control systems engineering and was chairman of IFAC Technical Committee of Adaptive and Learning Systems.
At Southampton he has established the Centre for Complex Autonomous Systems Engineering where he is now visiting professor. Today his main research interest is agent-based control systems and he is leading the Autonomous Systems and Robotics Group at the Department of Automatic Control at the University of Sheffield since 2013. He published about 200 papers, authored 4 books and co-authored numerous software packages.

Luke Muehlhauser: In "Autonomous Asteroid Exploration by Rational Agents" (2013), you discuss a variety of agent architectures for use in contexts where autonomous or semi-autonomous operation is critical — in particular, in outer space, where there are long communication delays between a robot and a human operator on Earth. Before we talk about rational agent architectures in particular, could you explain what an "agent" architecture is from your perspective, and why it is superior to other designs for many contexts?

Sandor Veres: First of all I would like to say that I am not sure that all kinds of agent architectures are "superior". Some agent paradigms have however advantages as a way of organizing your complex software system which controls an intelligent machine such as a robot. The feature I most like in some agent architectures is when they exhibit anthropomorphic type of operations so that they can handle statements about the world in terms of space and time, past and future, express possibilities and necessities, behaviour rules, knowledge of other agent's knowledge, including that of humans, can handle intentions and beliefs about a situation. If such constructs are part of the software, as opposed to being translated from a different kind of software, that makes things simpler in terms of programming. Also we would like robots to share with us their knowledge about the world, and we share our knowledge with them. It is an advantage if we share our ways of reasoning, a robot can be made more easy to understand and control if it is programmed in an anthropomorphic manner.

Luke: At one point in that paper you write that "Creating a new software architecture with significant benefits for autonomous robot control is a difficult problem when there are excellent software packages around…" What kinds of software architectures for autonomous robot control are out there already, and how generally applicable are they?

Sandor: By excellent packages I meant some robot operating system foundations such as the ROS for Linux and CCR for Windows on which one can build robot programs. Both of these have extensive libraries available for us to enable programming robot skills. For programming autonomous behaviour we have agent-oriented programming defined in exact terms by Yoav Shoham in 1993 and by now there are books just to review the many approaches to agent programming. There is also CLARAty by JPL which is continuous re-planning based rather than agent oriented. To summarise, GofAI has been transformed during the past 3 decades into various logic-based, subsumption, multi-layered and belief-desire-intention robot programming approaches, so there is a long and interesting history of great effort in software architectures for robots.

Luke: Where you think the future of autonomous agents is headed?

Sandor: I hope that soon not only agent programming languages will be available but standardised agents for various applications of robots. These can then be either programmed further or just simply trained further before deployed in an application. Eventually these agents could provide high level of integrity, capability and safety of robot operations autonomously. A next step will be to make these physically able agents truly social agents so that they behave in an appropriate manner and cooperate where required.

Luke: How might we get high-assurance autonomous agents for safety-critical applications?

Sandor: Determinism is a fundamental feature of digital computing so far, though we have random phenomena in some parallel computing we do on robots. Determinism of software has the advantage that our robots are also deterministic. We currently make every effort to make robots always respond in an appropriate manner, solve problems and be predictable despite often complex and disturbing environment. The more intelligent they are the more complex environment they can handle. The more complex environment however also means that it becomes more difficult to formally verify and test all their possible deterministic responses. So the trade off is between complexity of agent-software and "determinism" of intelligence, i.e. suitable actions by the robot at all times. Though the agent is deterministic, the sensors they use may not be reliable which make their response to some degree probabilistic. The challenge we face is hence to built up conceptual abstractions of complex environments which enable reliable decision making of our robotic agents. On the question of whether this is always possible in practice the jury is still out.

Luke: Will research progress on autonomy capabilities outpace research progress on safety research?

Sandor: This is a very good question and I believe this is likely to happen. High levels of capabilities will however reduce the probability of inappropriate response by a robot as long as it is accompanied by a formally verifiable decision making process of the agent. For agent development the most we can do is to make it "perfect", meaning that it should never intentionally do the wrong action or in case of failing hardware, it should always take the most likely positive action. This is easier said than done as the environment can create conflicting requirements for a robot's response and in such cases it needs to behave as a moral agent. Moral agents can use models of a broader context of a situation and apply principles over a wide range of knowledge. Moral agents of the future are likely to need the knowledge of an educated adult.

Luke: Thanks, Sandor!
The post Sandor Veres on autonomous agents appeared first on Machine Intelligence Research Institute.

Sandor Veres on autonomous agents


New Paper: "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem"

We've released a new paper recently accepted to the MIPC workshop at AAAI-14: "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem" by LaVictoire et al. This paper is essentially a shortened version of Barasz et al. (2014). For the history of the key results therein, see Robust Cooperation: A Case Study in Friendly AI Research.
Abstract of the new paper:
Applications of game theory often neglect that real-world agents normally have some amount of out-of-band information about each other. We consider the limiting case of a one-shot Prisoner's Dilemma between algorithms with read access to one anothers' source code. Previous work has shown that cooperation is possible at a Nash equilibrium in this setting, but existing constructions require interacting agents to be identical or near-identical. We show that a natural class of agents are able to achieve mutual cooperation at Nash equilibrium without any prior coordination of this sort.

The post New Paper: "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem" appeared first on Machine Intelligence Research Institute.

New Paper: “Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem”


Christof Koch and Stuart Russell on machine superintelligence

Recently, Science Friday (hosted by Ira Flatow) featured an interview (page, mp3) with Christof Koch and Stuart Russell about machine superintelligence. Christof Koch is the Chief Scientific Officer of the Allen Institute for Brain Science, and Stuart Russell is a computer science professor at UC Berkeley, and co-author of the world's most-used AI textbook.
I was glad to hear both distinguished guests take seriously the opportunities and risks of AGI. Those parts of the conversation are excerpted below:

Russell: Most [AI researchers] are like the Casters [from Transcendence]. They're just trying to solve the problem. They want to make machines intelligent. They're working on the next puzzle. There are [also] a number of people who think about what will happen when we succeed. I actually have a chapter in my book called, "What if we succeed?" If we make machines that are much smarter than human being, and this is what happens after Will Caster is uploaded, he has the massive computational resources to run his brain with, he becomes much more intelligent than humans.
If that happens, then it's very difficult to predict what such a system would do and how to control it. It's a bit like the old stories of the genie in the lamp or sorcerer's apprentice. You can give your 3 wishes, but in all those stories, the 3 wishes backfire, and there's always some loophole. The genie carries out what you ask to the utmost extent, and it's never what you really wanted. The same is going to be true with machines. If we ask them, "Heal the sick, end human suffering," perhaps the best way to end human suffering is to end human life altogether because then there won't be anymore human suffering. That would be pretty big loophole. You can imagine how hard it is to write tax laws so there are no loopholes. We haven't succeeded after 300 years of trying.
It's very difficult to say what we would want a super intelligent machine to do so that we can be absolutely sure that the outcome is what we really want as opposed to what we say. That's the issue. I think we, as a field, are changing, going through a process of realization that more intelligent is not necessarily better. We have to be more intelligent and controlled and safe, just like the nuclear physicist when they figured out chain reaction they suddenly realized, "Oh, if we make too much of a chain reaction, then we have a nuclear explosion." So we need controlled chain reaction just like we need controlled artificial intelligence.
Flatow: Isn't that true of all of science? The history of science whether we're talking about genetic engineering in its early days, wondering about what's going to crawl out of the lab once we start playing around the with the genome?
Koch: The difference now is that this [risk is an] existential threat to human society just like we'll see once a nuclear genie was out of the bottle, we still live under the possibility within 20 minutes that all people we know will be obliterated under nuclear mushroom. Now we live with the possibility that over the next 50, 100 years this invention that we're working on might be our final invention as Stuart has emphasized and that our future may be bleaker than we think.
[…]
Flatow: Do scientists have responsibilities to think about now the consequences of AI and are researchers organizing, talking, meeting about these things?
Russell: Absolutely. Yes. There's a responsibility because if you're working on a field whose success would probably be the biggest event in human history, and as some people predict, the last event in human history, then you'd better take responsibility. People are having meetings. In fact, I just organized one on Tuesday in Paris taking advantage of a major conference that was here. What I'm finding is that senior people in the field who have never publicly evinced any concern before are privately thinking that we do need to take this issue very seriously, and the sooner we take it seriously the better. I think the nuclear physicists wished they had taken it seriously much earlier than they did.
Flatow: Christof?
Koch: Yes. I fully have to agree, Stuart. I recently attended a meeting of physicists, and we had a large discussion and a poll about what are the biggest existential threat. To my surprise, after a nuclear war, it was the grey goo scenario or the AI run amok scenario. It wasn't climate control or some of the other more conventional ones. There is a lot of concern among some experts what will happen to run away AI.
[…]
There is no law in the universe that says things are limited to human level intelligence. There may well be entities that are going to be much smarter than us, and as Stuart has said, we have no way to predict what will be their desires, what will be their motivations, and programming it in, we all know how buggy software is. Do we really want to rely on some standard programmers to program in something that will not wipe us out in one way or another?
The post Christof Koch and Stuart Russell on machine superintelligence appeared first on Machine Intelligence Research Institute.


Exponential and non-exponential trends in information technology

Co-authored with Lila Rieber.
In The Singularity is Near, Ray Kurzweil writes that "every aspect of information and information technology is growing at an exponential pace."1 In Abundance, the authors list eight fields — including nanomaterials, robotics, and medicine — as "exponentially growing fields."2 The Second Machine Age says that "technical progress" in general is "improving exponentially."3
These authors are correct to emphasize that exponential trends in technological development are surprisingly common (Nagy et al. 2013), and that these trends challenge the wisdom of our built-in heuristic to ignore futures that sound absurd. (To someone in the 1980s, the iPhone is absurd. To us, it is an affordable consumer good.)
Unfortunately, these and other popular discussions of "exponential technologies" are often very succinct and therefore ambiguous, resulting in public and professional misunderstanding.4 I (Luke) regularly encounter people who have read the books above and come away with the impression that all information technologies show roughly exponential trends all the time. But this isn't true unless you have a very broad concept of what counts as "roughly exponential."
So, without speculating much about what Kurzweil & company intend to claim, we'll try to clear up some common misunderstandings about exponential technologies by showing a few examples of exponential and not-so-exponential trends in information technology. A more thorough survey of trends in information technology must be left to other investigators.5

Computations per dollar: still exponential
It's clear that Kurzweil himself does not literally mean that "every aspect of information and information technology is growing at an exponential pace," for he has previously discussed examples of non-exponential growth in some aspects of information technology. For example, he's well aware that the exponential trend in processor clock speed broke down in 2004, as shown in Fuller & Millett (2011a):6

Because this is a logarithmic chart, a straight line represents an exponential trend. Notice that clock speed stopped improving exponentially in 2004, but transistors per chip has continued to increase exponentially via the jump from single-core to multicore processors.
Elsewhere, Kurzweil tends to emphasize exponential trends in price-performance ratios specifically, for example computations per dollar. This is perfectly reasonable. Most of us don't care about the fine details of processor architecture — we just care about how much stuff we can do per dollar.7 And thus far, the exponential trend in computations per dollar has kept up.8
It's unclear, however, how much longer this trend can be maintained. In particular, the dark silicon problem may slow the currently exponential trend in computations per dollar. Joel Hruska covers other recent challenges to the trend here, including the halted production of the 450mm wafers that Intel, TSMC, and Samsung all bet their money on 18 months ago.

DRAM capacity per dollar: a slowing trend
Another important price-performance trend is DRAM capacity per dollar. This trend was almost precisely exponential for many years but recently the trend has slowed:9

The same slowing trend for DRAM is also reported in Hennessy & Patterson (2011), on page 17. On page 100 they remark:
DRAMs obeyed Moore's law for 20 years, bringing out a new chip with four times the capacity every three years. Due to the manufacturing challenges of a single-bit DRAM, new chips only double capacity every two years since 1998. In 2006, the pace slowed further, with the four years from 2006 to 2010 seeing only a [single] doubling of capacity.

SRAM capacity per dollar: a slowing trend
What about another kind of computer memory, like SRAM? The cost of SRAM dropped precipitously from 1980 to 1990 but has dropped more slowly since then.10

Hard drive capacity per dollar: interrupted by floods in Thailand
Cost per gigabyte of hard drive storage had been dropping exponentially for about 30 years when suddenly hard drive prices actually increased for a while because October 2011 floods in Thailand destroyed some hard drive factories. Hard drive prices have been comparatively flat since then:11

The floods hurt many computing providers, who struggled (and sometimes failed) to offer services at the low prices they had anticipated based on exponential expectations: see e.g. the comments by BackBlaze, Intel, and Joyent.
Whereas the exponential trend for processor clock speed was brought to a halt by physics, the exponential trend in cost per gigabyte of storage was slowed (at least for now) by natural disaster.
Page 85. In the same book, he also writes that "we see ongoing exponential growth of every aspect of information technology, including price-performance, capacity, and rate of adoption." (p. 377). In How to Create a Mind, Kurzweil writes that "In the course of my investigation, I made a startling discovery: If a technology is an information technology, the basic measures of price/performance and capacity… follow amazingly precise exponential trajectories" (p. 254).Page 57. In general, Diamandis & Kotler seem to agree with Kurzweil that all information technologies experience exponential growth curves. E.g. on page 99 they write that "Although [some agroecological] practices themselves look decidedly low tech, all the fields they're informed by are information-based sciences and thus on exponential growth curves," and on page 190 they write that "almost every component of medicine is now an information technology and therefore on an exponential trajectory."Page 10. The authors also seem to expect exponential trends for anything that becomes a digital process: "…batteries… haven't improved their performance at an exponential rate because they're essentially chemical devices, not digital ones…" (p. 52).E.g. Kurzweil seems to use a fairly loose definition of "exponential." For example in Kurzweil (2001) he gives this chart of ISP cost-performance as an example exponential trend. Sometimes this seems to cause confusion in dialogue. For example, in response to Ilkka Tuomi's criticisms (2002, 2003) of claims of exponential trends in computing, Kurzweil wrote that if Tuomi were correct, "I would have to conclude that the one-quarter MIPS computer costing several million dollars that I used at MIT in 1967 and the 1000 MIPS computer that I purchased recently for $2,000 never really existed… I admire his tenacity in attempting to prove that the world of information technology is flat (i.e., linear)." But Tuomi's views don't entail that, and Tuomi didn't say that trends in information technology have been linear. The conflict appears to stem from the fact that Tuomi was using "exponential" in the strict sense, while Kurzweil was using the term in a very loose sense. This becomes clearer in Tuomi's reply to Kurzweil.Our thanks to Jonah Sinick for his assistance in researching this post.It should be noted, however, that the exponential trend line for clock speed on page 61 of The Singularity is Near (2005) is now known to be incorrect. Kurzweil's graph used the 2002 ITRS report to project the trend line for 2001-2016, but actual growth in clock speed fell substantially short of the ITRS projection.As Fuller & Millett (2011b) write, "When we talk about scaling computing performance, we implicitly mean to increase the computing performance that we can buy for each dollar we spend" (p. 81).Kurzweil (2012), ch. 10, footnote 10 shows "calculations per second per $1,000" growing exponentially from 1900 through 2010, including several data points after 2004. However, we couldn't find his data sources, and we don't know whether he adjusted for inflation, so we've relied instead on a data set provided by Koh & Magee (2006), extended by data we pulled from NotebookCheck, PCStats, Tom's Hardware, and CPU-World. Our raw data are here and show a continuing exponential trend in MIPS per dollar, adjusted for inflation.This chart uses data from Bryant & O'Hallaron (2011), p. 584 and from the Performance Curve Database. Raw data here.Data from Bryant & O'Hallaron (2011), p. 584. Raw data here.Our data are drawn from Matthew Komorowski's page on storage cost (but adjusted for inflation), Koh & Magee (2006), Bryant & O'Hallaron (2011), and the Performance Curve Database.The post Exponential and non-exponential trends in information technology appeared first on Machine Intelligence Research Institute.


Benjamin Pierce on clean-slate security architectures

 Benjamin C. Pierce is Henry Salvatori Professor of Computer and Information Science at the University of Pennsylvania and a Fellow of the ACM. His research interests include programming languages, type systems, language-based security, computer-assisted formal verification, differential privacy, and synchronization technologies. He is the author of the widely used graduate textbooks Types and Programming Languages and Software Foundations.
He has served as co-Editor in Chief of the Journal of Functional Programming, as Managing Editor for Logical Methods in Computer Science, and as editorial board member of Mathematical Structures in Computer Science, Formal Aspects of Computing, and ACM Transactions on Programming Languages and Systems. He is also the lead designer of the popular Unison file synchronizer.

Luke Muehlhauser: I previously interviewed Greg Morrisett about the SAFE project, and about computer security in general. You've also contributed to the SAFE project, gave an "early retrospective" talk on it (slides), and I'd like to ask you some more detailed questions about it.
In particular, I'd like to ask about the "verified information-flow architecture" developed for SAFE. Can you give us an overview of the kinds of information flow security properties you were able to prove about the system?

Benjamin C. Pierce: Sure. First, to remind your readers: SAFE is a clean-slate design of a new hardware / software stack whose goal is to build a network host that is highly resilient to cyber-attack. One pillar of the design is pervasive mechanisms for tracking information flow. The SAFE hardware offers fine-grained tagging and efficient propagation and combination of tags on each instruction dispatch. The operating system virtualizes these generic facilities to provide an "information-flow abstract machine," on which user programs run.
Formal verification has been part of the SAFE design process right from the beginning. We'd originally hoped to be able to verify the actual running code of the OS in the style of sel4, but we found that the codebase was too big and moving too fast for this to be practical with a small verification team. Instead, we've developed a methodology that combines full formal verification of models of the system's key features with "property-based random testing" (à la QuickCheck) of richer subsets of the system's functionality.
Our most interesting formal proof so far shows that the key security property of the information-flow abstract machine — the fact that a program's secret inputs cannot influence its public outputs — is correctly preserved by our implementation on (a simpified version of) the SAFE hardware. This is interesting because the behavior of the abstract machine is achieved by a fairly intricate interplay between a hardware "rule cache" and a software layer that fills the cache as needed by consulting a symbolic representation of the current security policy. Since this mechanism lies at the very core of the SAFE architecture's security guarantees, we wanted to be completely sure it is correct (and it is!).

Luke: You mention that since you aren't able to formally verify the entire OS with your available resources, you're instead formally verifying models of the system's key features and also using property-based random testing of some of the rest of the system's functionality. What level of subjective confidence (betting odds) does that approach give you for the security of the system, compared to what you could get if you had the resources to successfully verify the entire OS? (A rough "gut guess" is acceptable!)

Benjamin: Let's say that an unverified but well-engineered and road-tested system gets a 1 on the security scale and a fully verified OS gets an 8. And let's assume that we're talking about a finished and polished version of SAFE (recognizing that what we have now is an incomplete prototype), together with correctness proofs for models of core mechanisms and random testing of some larger components. My gut would put this system at maybe a 4 on this scale. The proofs we've done are limited in scope, but they've been extremely useful in deepening our understanding of the core design and guiding us to a clean implementation.
Why is the top of my scale 8 and not 10? Because even a full-blown machine-checked proof of correctness doesn't mean that your OS is truly invulnerable. Any proof is going to be based on assumptions — e.g., that the hardware behaves correctly, which might not be the case if the attacker has a hair dryer and physical access to the CPU! So, to get higher than 8 requires some additional assurance that the degree of potential damage is limited even if your formal assumptions are broken.
This observation suggests a defense-in-depth strategy. Our goal in designing the SAFE software has been to make the "omniprivileged" part of the code (i.e., the part that could cause the machine to do absolutely anything, if the attacker could somehow get it to misbehave) as small as possible — vastly smaller than conventional operating systems and even significantly smaller than microkernels. The code that checks security policy and fills the hardware rule cache is highly privileged, as is the garbage collector in our current design, but we believe that the rest of the OS can be written as a set of "compartments" with more limited privilege, using the tagging hardware to enforce separation between components. (Demonstrating this claim is ongoing work.)

Luke: I've spoken to some people who are skeptical that formal verification adds much of a security boost, since so many security problems come from not understanding the vulnerabilities well enough to capture them all in a system's formal requirements in the first place, as opposed to coming from a system which fails to match the design specification in ways that could have been caught by formal verification. Yet you seem to rank a verified system as far more secure than a non-verified system. What's your perspective on this?

Benjamin: Yes, other things being equal I would put quite a bit of money on a verified system being more secure than an unverified one — not because a the verification process eliminates the possibility of vulnerabilities that were not considered in the specification, but because the process of writing the formal specification requires someone to sit down and think carefully about at least some of the (security and other) properties that the system is trying to achieve. A clear understanding of at least some of the issues is a lot better than nothing!
Actually carrying out a proof that the system matches the specification adds significant further value, because specifications themselves are complex artifacts that require debugging just as much as the system being specified does! But property-based random testing — i.e., checking that the system satisfies its specification for a large number of well-distributed random inputs — can be a very effective intermediate step, giving a large part of the bang for relatively small buck.

Luke: In general, do you think that highly complex systems for which "high assurance" of safety or security (as relevant) are required need to be designed "from the ground up" for safety or security in order to obtain such high assurance, or are there many cases in which a highly complex but high assurance system can be successfully developed by optimizing development for general performance and then "patching" the system for security and safety?
And, do you have any guesses as to what the most common reply to this question might be in your own subfield(s) and other adjacent subfields?

Benjamin: Of course it depends a bit what you mean by "high assurance"! Quite a few people define the term so that it only applies to software developed using methods that consider safety and/or security from day 1. But even if we take a broader definition of HA — "offering very good resistance to malicious penetration and catastrophic failure," or some such — my feeling is that this is extremely difficult to retrofit.
I personally saw this effect very clearly while building Unison, a cross-platform file synchronization tool that I made with Trevor Jim and Jérome Vouillon a few years ago. I had done several synchronizers before Unison, but I always discovered edge cases where the behavior seemed wrong, so we began the Unison design by trying to write down a very precise (though dramatically simplified) specification of its core behavior. Beyond helping us understand edge cases, this exercise had a huge effect on the way we wrote the code. For example, the specification says that, at the end of a run of the synchronizer, every part of the filesystems being synchronized should be either synchronized (brought up to date with the other copy) or unchanged from when the synchronizer started running (for example, if there was a conflict that the user chose not to resolve). But, since the synchronizer can terminate in all sorts of ways (finishing normally, getting killed by the user, the network connection getting dropped, one machine getting rebooted, …) this requirement means that the filesystems must be in one of these two states at every moment during execution. This is actually quite hard to achieve: there are many parts of the code that have to do extra work to buffer things in temporary locations until they can be moved atomically into place, use two-phase commits to make sure internal data structures don't get corrupted, etc., etc., and the reasons why the extra work is needed are often quite subtle. Having the spec beforehand forced us to do this work as we went. Going back later and rediscovering all the places where we should have paid extra attention would have been well nigh impossible.
As for what others in my research area might say to your question, I decided to poll some friends instead of just assuming. As expected, most of the comments I got back were along the lines of "Making a complex system really secure after the fact is basically impossible." But a few people did propose systems that might count as examples, such as SELinux and Apache. (Actually, SELinux is an interesting case. On one hand, its goal was only to add mandatory access control to Linux; it explicitly didn't even attempt to increase the assurance of Linux itself. So for attacks that target kernel vulnerabilities, SELinux is no better than vanilla Linux. On the other hand, I'm told that, in certain situations — in particular, for securing web applications and other network-facing code against the sorts of attacks that come over HTTP requests — SELinux can provide very strong protection.)

Luke: Thanks, Benjamin!
The post Benjamin Pierce on clean-slate security architectures appeared first on Machine Intelligence Research Institute.


Michael Fisher on verifying autonomous systems

 Michael Fisher is a professor of Computer Science, specialising in logical methods and automated formal verification, and Director of the multi-disciplinary Centre for Autonomous Systems Technology at the University of Liverpool and member of the Logic and Computation group in the Department of Computer Science.
Professor Fisher is also a Fellow of both the BCS and the IET, and a member of the UK Computing Research Committee. He is Chair of the Department's Industrial Liaison Committee.

Luke Muehlhauser: In "Verifying Autonomous Systems" (2013), you and your co-authors summarize some recent approaches to formally verifying autonomous systems. At one point, you write:
Many autonomous systems, ranging over unmanned aircraft, robotics, satellites, and even purely software applications, have a similar structure, namely layered architectures as summarized in Figure 1. Although purely connectionist/sub-symbolic architectures remain prevalent in some areas, such as robotics, there is a broad realization that separating out the important/difficult choices into an identifiable entity can be very useful for development, debugging, and analysis. While such layered architectures have been investigated for many years they appear increasingly common in autonomous systems.
The accompanying figure is:

What can you say about how common you think something like your "typical hybrid autonomous system architecture" is? E.g. do you think that kind of architecture is in use in ~20 different real-world applications, or ~200 different real-world applications, or perhaps more? Do you think there's a trend in favor of such architectures? Is there a trend toward such architectures even in robotics, do you think?

Michael Fisher: It is difficult to get definitive evidence about the internal architectures of autonomous systems, especially commercial ones, so I only have anecdotal information about systems constructed in this way. However, from my discussions with engineers, both in academia and industry (aerospace and automotive), it appears that the use of this sort of architecture is increasing. It might not be termed an "agent architecture", indeed the "decision maker" might not be termed an "agent" at all, but systems with a separate "executive" or "control module" that explicitly deals with high-level decisions, fit into our model.
I am not sure whether robotic architectures are also moving in this direction, but there seems to be a theme of adding a "monitor" or "governor" that can assess the safety or ethics of proposed actions before the robot takes them1, 2 . This can alternatively be viewed as an additional agent making high-level decisions (what to permit, what to block) and so this approach might also be seen as a variety of hybrid agent architecture.

Luke: In that same paper, you and your co-authors explain that
Since autonomous software has to make its own decision, it is often vital to know not only what the software does and when it does it but also why the software chooses to do it.
…A very useful abstraction for capturing such autonomous behavior within complex, dynamic systems turns out to be the concept of an agent. Since the agent concept came into prominence in the 1980s, there has been vast development within both academia and industry. It has become clear this agent metaphor is very useful for capturing many practical situations involving complex systems comprising flexible, autonomous, and distributed components. In essence, agents must fundamentally be capable of flexible autonomous action.
However, it turns out the "agent" concept on its own is still not enough! Systems controlled by neural networks, genetic algorithms, and complex control systems, among others, can all act autonomously and thus be called agents, yet the reasons for their actions are often quite opaque. Because of this, such systems are very difficult to develop, control, and analyze.
So, the concept of a rational agent has become more popular. Again, there are many variations but we consider this to be an agent that has explicit reasons for making the choices it does, and should be able to explain these if necessary.
I previously explored the relative transparency of different AI methods in Transparency in Safety-Critical Systems, where I asked:
How does the transparency of a method change with scale? A 200-rules logical AI might be more transparent than a 200-node Bayes net, but what if we're comparing 100,000 rules vs. 100,000 nodes? At least we can query the Bayes net to ask "what it believes about X," whereas we can't necessarily do so with the logic-based system.
What are your thoughts on how well the transparency of rational agent models will scale? Or, do you think the "rational agent" part of future autonomous systems will be kept fairly small (e.g. to enable transparency and verification) even as the subsystems at lower "layers" in the architecture may grow massively in complexity?

Michael: I would say that the high-level rules within rational agents remain relatively transparent even as their numbers increase. It's just that the proof/analysis mechanisms do not scale so well.
So, as you suggest, the intention is to keep the rational agent part as small as possible. Formal verification is quite costly, so any verification of very large agents is likely to be infeasible.
By only dealing with the high-level, rational agent decisions, we aim to avoid handling the vast state-space of the whole system. The use of logical (specifically, BDI) rules abstracts us from much low-level detail.
Concerning your question about "100,000 rules vs. 100,000 nodes", I would hope that the symbolic logic language used helps us reduce this to 1000, or even 100, rules (vs. 100,000 nodes).

Luke: Does your hierarchical autonomous system approach, with a Belief-Desire-Intention rational agent "on top," integrate with hybrid system control approaches, or would it be an alternative to hybrid system control approaches?

Michael: Yes, it fits together with hybrid system approaches and, certainly, the continuous and stochastic aspects are important for modelling real-world interactions. You can possibly see our approach as providing further detail about the discrete steps within hybrid systems.

Luke: In the future, do you think the logical agent specification in the top-level role in hierarchical autonomous systems will eventually need to be replaced by a more flexible, probabilistic/fuzzy reasoning approach, so as to be a better match for the environment? Or could a logical agent specification play that role even as we build systems (several decades from now) that approach human-like, fully general reasoning and planning abilities?

Michael: The latter. We are working towards being able to specify and verify probabilistic, and even real-time, agent behaviours at this top level. (Currently we can do either agent verification or probabilistic verification, but we are developing combination mechanisms.) The high-level agent will likely become increasingly complex as the sophistication of applications develop and may well incorporate different reasoning and decision components. Our approach at present is that planning components are within the separate, low-level, modules. The agent just calls these and then makes decisions between the plan options produced.

Luke: Interesting. Could you say a bit more about the combination mechanisms you're developing?

Michael: Formally verifying basic logical properties is already difficult. However, once we tackle autonomous systems that might well be used in realistic situations, then we are often led to much more complex logical queries. For example, imagine we wish to verify a property such as
when asked, the probability of the robot replying within 30 seconds is at least 80%
Already we have moved beyond basic properties to incorporate both probabilities (>80%) and real-time aspects (within 30s). But, with autonomous systems we would like to verify not just what the system (e.g. a robot) does but what its intentions and beliefs are. So we might even want to verify
when asked, the probability of the robot believing the request has been made and then intending to reply within 30 seconds is at least 80%
The problem here is that the logical framework we need to describe such requirements is already quite complex. Combinations of logics are often difficult to deal with and combinations of logics of probability, real-time, intention, belief, etc., have not yet been studied in great detail.
We are aiming to develop and extend some theoretical work we have been doing on model checking of complex logical combinations in order to tackle this problem of more comprehensive autonomous system verification.

Luke: Thanks, Michael!
Arkin, R. "Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture". Technical Report GIT-GVU-07-11, Georgia Tech, 2007.Woodman, R., Winfield, A., Harper, C. and Fraser, M. "Building Safer Robots: Safety Driven Control". International Journal of Robotic Research 31(13):1603–1626, 2012.The post Michael Fisher on verifying autonomous systems appeared first on Machine Intelligence Research Institute.