2013 in Review: Friendly AI Research

This is the 4th part of my personal and qualitative self-review of MIRI in 2013, in which I review MIRI's 2013 Friendly AI (FAI) research activities.1
Friendly AI research in 2013

In early 2013, we decided to shift our priorities from research plus public outreach to a more exclusive focus on technical FAI research. This resulted in roughly as much public-facing FAI research in 2013 as in all past years combined.
Also, our workshops succeeded in identifying candidates for hire. We expect to hire two 2013 workshop participants in the first half of 2014.
During 2013, I learned many things about how to create an FAI research institute and FAI research field. In particular…
MIRI needs to attract more experienced workshop participants.
Much FAI research can be done by a broad community, and need not be labeled as FAI research. But, more FAI progress is made when the researchers themselves conceive of the research as FAI research.
Communication style matters a lot.

The shift to Friendly AI research
From MIRI's founding in 2000 until our strategic shift in early 2013,2 we did some research and much public outreach (e.g. the Singularity Summit and The Sequences).3 In early 2013, we decided that enough outreach and movement-building had been done that we could productively shift to a primary focus on research, and Friendly AI research specifically.
The task before us was, essentially, to create a new FAI research institute (out of what had previously been primarily an outreach organization), and to create a new field of FAI research. We still have much to learn about how to achieve these goals (see below).
Our initial steps were to (1) hold a series of research workshops, and to (2) describe open problems in Friendly AI theory to potential research collaborators. Our workshops and open problem descriptions were aimed at three goals in particular. We wanted them to:

help us identify researchers MIRI should hire to work full-time on Friendly AI theory,
expose additional researchers to the Friendly AI research agenda, and
spur concrete progress on open problems in Friendly AI.

First, I'll describe our 2013 Friendly AI research activities. After that, I'll review "how good" I think these results are, and what lessons I've learned.
The workshops
The workshops strategy had been suggested by the success of our one-week November 2012 workshop, which had been an experiment involving only four researchers, and had produced the core result of "Definability of Truth in Probabilistic Logic."
Our first workshop of 2013, held in April, was an attempt to tackle as many open problems as we could, with as many people as we could gather, to quickly learn which problems were most tractable and which researchers were most likely to contribute in the future. It involved 12 participants and lasted 3 weeks, though (due to scheduling constraints) only 5 researchers participated for the entire duration of the workshop. We learned a great deal about the workshop's participants, and three problems in particular showed the most progress: Christiano's "Definability of Truth" framework, LaVictoire's "Robust Cooperation" framework, and Fallenstein's "parametric polymorphism" approach to the Löbian obstacle for self-modifying systems. The success of this workshop encouraged us to hold more such workshops, albeit at a smaller scale and with tighter research foci.
Our next workshop, in July 2013, had 8 participants and lasted one week. It focused on issues related to logical omniscience and the Löbian obstacle / self-reflective agents, and produced less progress-per-day than the April workshop. Its chief result was described in a blog post by participant Abram Demski.
Our September workshop focused instead on decision theory. It had 11 participants and lasted one week. Participants brainstormed "well-posed problems" in the area, built on LaVictoire's robust cooperation framework, made some progress on formalizing updateless decision theory, and formulated additional toy problems such as the Ultimate Newcomb's Problem.
Our November workshop was our first workshop held outside of Berkeley. FHI graciously hosted us at Oxford University. As with the July workshop, this workshop focused on logical omniscience and self-reflective agents. There were 11 participants, and it lasted one week. November's theoretical progress flowed into the progress made at our December workshop (same topics, 13 participants, one week), which was captured in 7 new technical reports.
Next, some basic statistics:

We held 5 research workshops in 2013, with all but one of them being one week long.
These workshops were attended by 35 unique researchers, plus 7 first-day-only visitors (e.g. Hannes Leitgeb and Nik Weaver).4
For first-time attendees, the median reply to the question "How happy are you that you came to the workshop, 0-10?" was 8.5.
From the time it went live in March 2013 to the end of 2013, about a dozen people contacted us about our Recommended Courses for MIRI Math Researchers page. However, we have reason to believe it has influenced the study patterns of a much larger number of people. Some MIRI supporters have told us they routinely point smart young acquaintances to that page. Moreover, the page received more unique pageviews in 2013 than (e.g.) our Donate or About pages, despite not being linked from every page of the site like the Donate and About pages are. The Recommended Courses page made it possible for at least one person (Nate Soares) to quickly upgrade his math skills and attend a workshop in 2013, which he couldn't have done before studying several of the textbooks on the Courses page.
From the time it went live in June 2013 to the end of 2013, we received 227 non-junk applications5 to attend future MIRI workshops, 47 of which are still being processed. So far, 60 applicants are ones we've deemed "promising," 23 of whom attended a workshop in 2013. Of those 23, about half were researchers with whom we had little to no prior contact.

Describing open problems in Friendly AI
In 2013, MIRI described open problems in Friendly AI (OPFAIs) to researchers via three standard methods: articles, talks, and tutorials at workshops.
On OPFAI articles: Yudkowsky's article on "OPFAI #1" discussed intelligence explosion microeconomics (aka AI takeoff dynamics), which I consider to be an open problem in "strategy research" rather than in Friendly AI theory, so I discussed it in a previous post. From my perspective, the first written OPFAI description of 2013 was on logical decision theory. Alex Altair (then a MIRI researcher) described the problem in an April 2013 paper called "A Comparison of Decision Algorithms on Newcomblike Problems." This open problem had been described before, in Less Wrong posts and in a 117-page technical report, but Altair's presentation of the issue was more succinct and formal than previous presentations had been.
The second written OPFAI description of 2013 was on the tiling agents problem, and specifically the Löbian obstacle to tiling agents. Yudkowsky brought a draft of this paper to the April workshop, and heavily modified the draft as a result of the progress at that workshop, finally publishing the draft in June 2013. The third written OPFAI description of 2013, by Patrick LaVictoire and co-authors, was on the robust cooperation problem. The fourth written OPFAI description of 2013 was on naturalized induction.
Because the tiling agents paper took ~2 months of FAI researcher time to produce, we decided to experiment with a process that would minimize the amount of FAI-researcher-time required to produce new OPFAI descriptions. First, Yudkowsky brain-dumped the OPFAI to a Facebook group. Then, Robby Bensinger worked with several others to produce Less Wrong posts that described the OPFAI more clearly. The first post produced via this process was published in December 2013: Building Phenomenological Bridges. The rest of the posts explaining this OPFAI will be published in Q1 2014. Because we want to maximize the amount of FAI researcher hours that goes into FAI research rather than exposition, we hope to hire additional expository writing talent in 2014 (see our Careers page).
On OPFAI talks: MIRI scheduled two OPFAI talks in 2013. Yudkowsky's Oct. 15th talk, "Recursion in rational agents: Foundations for self-modifying AI," described both the robust cooperation and tiling agents problems to an audience at MIT. Two days earlier, (MIRI research associate) Paul Christiano gave a talk about probabilistic metamathematics at Harvard, following up on the earlier results from the "Definability of Truth" paper.6 Unfortunately, Yudkowsky's talk was not recorded, but Christiano's was.
On OPFAI tutorials at workshops: Each MIRI workshop in 2013 opened with a day or two of tutorials on the open problems being addressed by that workshop. These tutorials exposed ~35 researchers (participants and first-day visitors) to OPFAIs they weren't previously very familiar with. (The others — e.g. Yudkowsky, Christiano, and Fallenstein — were already pretty familiar with the OPFAIs described in the tutorials.)
How good are these results?
For comparison's sake, MIRI's 2000-2012 FAI research efforts consisted in:

Yudkowsky's early research into the general "shape" of the Friendly AI challenge, resulting in publications such as "Creating Friendly AI" (2001), "Coherent Extrapolated Volition" (2004), and "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (2008). These publications did not yet describe any OPFAIs as well-defined as the open problems described in Altair (2013), Yudkowsky & Herreshoff (2013), or LaVictoire et al (2013).7
Yudkowsky's early decision theory research, which resulted in TDT circa 2005, though this work wasn't written up in much detail until 2009 (1, 2, 3) and 2010.
Yudkowsky's early work on Friendly consequentialist AI, in 2003-2009, some of it with Marcello Herreshoff, and one summer (2006) with Peter de Blanc and Nick Hay as well. This work resulted in early versions of many of the OPFAIs described by MIRI in 2013, currently being written up, or currently in Yudkowsky's queue to write up. It also resulted in the "infinite waterfall" method later described in Yudkowsky & Herreshoff (2013).
Yudkowsky worked again with Herreshoff in the summer of 2009, in part on the Löbian obstacle.
MIRI held a decision theory workshop in March 2010, attended by Eliezer Yudkowsky, Wei Dai, Stuart Armstrong, Gary Drescher, Anna Salamon, and about a dozen others who were present for some but not all discussions.8 This workshop spawned a decision theory mailing list that has, from 2010 through the present day, produced much of the recent progress on TDT/UDT-style decision theories, though mostly via non-MIRI researchers like Wei Dai, Vladimir Slepnev, Stuart Armstrong, and Vladimir Nesov.
(Former MIRI researcher) Peter de Blanc's work on "convergence of expected utility for universal AI" and ontological crises, resulting in de Blanc (2009) and de Blanc (2011).
(MIRI research associate) Daniel Dewey's work on value learning, resulting in Dewey (2011).

Thus, MIRI's public-facing Friendly AI research from 2000-2012 consisted in a few non-technical works like "Creating Friendly AI" and "Coherent Extrapolated Volition," some philosophical writings on TDT, and three somewhat technical papers by Peter de Blanc and Daniel Dewey. Compare this to MIRI's 2013 public-facing FAI research: Muehlhauser & Williamson (2013),9 Altair (2013), Christiano et al. (2013), Yudkowsky & Herreshoff (2013), LaVictoire et al (2013), and these 7 technical reports.10
Subjectively, it feels to me like MIRI produced about as much public-facing Friendly AI research progress in 2013 as in all past years combined (2000-2012), and possibly more. This is good but not particularly surprising, since 2013 was also the first year in which MIRI tried to focus on producing public-facing FAI research progress. (But to be clear: if we remove the "public-facing" qualifier, then it's clear that Yudkowsky alone produced far more FAI research progress in 2000-2012 than MIRI and its workshops produced in 2013 alone.)
So, did our workshops and open problem descriptions achieve our stated goals? Let's check:

Yes, they helped us identify candidates for hire. We expect to hire two 2013 workshop participants in the first half of 2014. (One of these hires is pending a visa application approval.)
Yes, they exposed many new researchers to the Friendly AI research program. But, this exposure didn't lead to as much independent Friendly AI work as I had hoped, and I have some theories as to why this was (see below).
Yes, they spurred concrete research progress on Friendly AI (see above).

While this represents a promising start toward growing an FAI research institute and a new field of FAI research, there are many dimensions on which our output needs to improve for MIRI to have the impact we hope for (see below).
What have I learned about how to create an FAI research institute, and a new field of FAI research?
Some of my "lessons learned" from 2013's FAI research activities were things I genuinely didn't know at the start of the year. Most of them are things I suspected already, and I think they were confirmed by our experiences in 2013. Here are a few of them, in no particular order.
1. Keep operations work away from researchers.
In other words, "Don't be afraid of a high operations-staff-to-researchers ratio." Operations talent (including executive talent) is easier to find than FAI research talent, so it's important to hire sufficient operations talent to make sure the FAI researchers we do find can spend approximately all their time on FAI research, and almost none of their time on tasks that can mostly be handled by operations staff (writing grant proposals, organizing events, fundraising, paper bibliographies, etc.). MIRI should hire enough operations talent to do this even if it makes our operations-staff-to-researcher ratio looks high for a research institute.11
Universities often struggle with this (from a research productivity perspective), loading up some of the best research talent in the world with teaching duties, grant writing duties, and university service.12 As an independent research institute, MIRI can set its own policies and minimize these problems.
2. We need to attract more experienced workshop participants.
Our workshops attracted some very bright participants, but they were almost exclusively younger than 30, with relatively few publications to their name. More experienced researchers would probably have advantages in (1) knowing related results and formal tools, (2) knowing productive research tactics, and (3) writing up results for peer-review, among other advantages.
3. Much FAI research can be done by a broad community, and need not be labeled as FAI research.
Presently, the Yudkowskian paradigm for "Friendly AI research" describes a very large research program that breaks down into dozens of sub-problems (OPFAIs), e.g. the tiling agents toy problem. Locating and formulating open problems plausibly relevant for Friendly AI is a challenge in itself, one that especially benefits from specializing in Friendly AI for several years.
Many of the OPFAIs themselves, however, can be framed as "ordinary" open problems in AI safety engineering, philosophy, mathematical logic, theoretical computer science, economics, and other fields. These open problems can often be stated without any mention of Friendly AI, and sometimes without any mention of AI in general.
For every OPFAI Yudkowsky has described,13 I've been able to locate earlier related work.14 Although this earlier work has not produced what we would regard as good solutions to open problems in FAI, it does suggest that FAI can be framed in ways palatable to academia. FAI need not be an "alien" research program, operating strictly outside mainstream academia, and conducted only by those explicitly motivated by FAI. Instead, FAI researchers should be able to frame their work in the context of mainstream research paradigms if they choose to do so. Moreover, much FAI research can be done even by those who aren't explicitly motivated by FAI, so long as they find (e.g.) the Löbian obstacle interesting as mathematics — or as computer science, or as philosophy, etc.

4. But, more FAI progress is made when the researchers themselves conceive of the research as FAI research.
Still, researchers do seem more likely to produce useful work on Friendly AI if they are thinking about the problems from the perspective of Friendly AI, rather than merely thinking about them as interesting open problems in philosophy, computer science, economics, etc. As I said in my conversation with Jacob Steinhardt:
People work on different pieces of the problem depending on whether they're trying to solve the problem for Friendly AI or just for a math journal. If they aren't thinking about it from the FAI perspective, people can work all day on stuff that's very close to what we care about in concept-space and yet has no discernable value to FAI theory. Thus, the people who have contributed by far the most novel FAI progress are people explicitly thinking about the problems from the perspective of FAI…

5. Communication style matters a lot.
When I talk to the kinds of top-notch researchers MIRI would like to collaborate with on open problems in Friendly AI, perhaps the most common complaint I hear is that our work is not formal enough, or not described clearly enough for them to understand it without more effort on their part than they are willing to expend. For an example of such a conversation that was recorded and transcribed, see again my conversation with Jacob Steinhardt.
I've thought this for a long time, and my experiences in 2013 have only reinforced the point. I'll be writing more about this in the future.

What counts as "Friendly AI research" is, naturally, a matter of debate. For most of this post I'll assume "Friendly AI research" means "what Yudkowsky thinks of as Friendly AI research," with the exception of intelligence explosion microeconomics, for reasons given in this post.Until early 2013, the organization currently named "Machine Intelligence Research Institute" was known as the "Singularity Institute for Artificial Intelligence."From 2000-2004, "MIRI" was just Eliezer Yudkowsky, doing early FAI research. The organization began to grow in 2004, and by 2006 most efforts were outreach-related rather than research-related. This remained true until early 2013.Some statistics about 2013's 35 workshop participants: 15 have a PhD, three are women, and 3 hold a university faculty position of assistant professor or higher rank. In short, our workshop participants have thus far largely been graduate students, post-docs, and independent researchers. Among the 15 participants who have a PhD, 9 have a PhD in mathematics, 4 have a PhD in computer science, one has a PhD in cognitive science, and one has a joint PhD in philosophy and computer science.By "junk applications" I mean to include both spam applications and applications from people who are clearly incapable of math research, e.g. "Hello, I would love to come to America to learn algebra."Probabilistic metamathematics is an OPFAI in itself, and also one possible path toward a solution to the tiling agents problem.The open problems in these publications, too, need additional formalization. Such is the current state of research.For example, Steve Rayhawk and Henrik Jonsson.This short paper lies deep in the "philosophy" end of the philosophy -> math -> engineering spectrum.For both the 2000-2012 and 2013 calendar periods, when I write of "MIRI's public-facing FAI work" I'm not including work that was "enabled" but not really "produced" by MIRI or its workshops, for example most work on UDT/ADT (which were nevertheless largely developed on MIRI's LessWrong.com website and its decision theory mailing list).At the end of 2013, we had five full-time staff members: Luke Muehlhauser (executive director), Louie Helm (deputy director), Eliezer Yudkowsky (research fellow), Malo Bourgon (program manager), and Alex Vermeer (program management analyst), totaling 4 operations staff and one researcher. This 4:1 ratio will shrink as we are able to hire more FAI researchers, but I think it would have been a mistake to try to get by with fewer operations staff in 2013.Link et al. (2008); Marsh & Hattie (2002); NSOPF (2004).Sometimes with much help from Robby Bensinger and/or others.I'll list some examples of earlier related work. (1) Superrationality: getting agents to rationally cooperate with agents like themselves. Before "Robust Cooperation" there was: Rapoport (1966); McAfee (1984); Hofstadter (1985); Binmore (1987); Howard (1988); Tennenholtz (2004); Fortnow (2009); Kalai et al. (2010); Peters & Szentes (2012). (For Rapoport 1966, see especially pages 141-144 and 209-210. (2) Coherent extrapolated volition: figuring out what we would wish if we knew more, thought better, were more the people we wished we were, etc. Before Yudkowsky (2004) there was: Rawls (1971); Harsanyi (1982); Railton (1986); Rosati (1995). (For an overview of this background, see Muehlhauser & Williamson 2013.) (3) Parliamentary methods for values aggregation: using voting mechanisms to resolve challenges in normative uncertainty and values aggregation. Before Bostrom (2009) there was a vast literature on this topic in social choice theory. For recent overviews, see List (2013); Brandt et al. (2012); Rossi et al. (2011); Gaertner (2009). (4) Reasoning under fragility: figuring out how to get an agent not to operate with full autonomy before it has been made fully trustworthy. Before Yudkowsky began to discuss the issue, there was much work on "adjustable autonomy": Schreckenghost et al. (2010); Mouaddib et al. (2010); Zieba et al. (2010); Pynadath & Tambe (2002); Tambe et al. (2002). (5) Logical decision theory: finding a decision algorithm which can represent the agent's deterministic decision process. Before Yudkowsky (2010) there was: Spohn (2003); Spohn (2005). (6) Stable self-improvement: getting a self-modifying agent to avoid rewriting its own code unless it has very high confidence that these rewrites will maintain desirable agent properties. Before Yudkowsky & Herreshoff (2013) there was: Schmidhuber (2003); Schmidhuber (2009); Steunebrink & Schmidhuber (2012). (7) Naturalized induction: getting an induction algorithm to treat itself, its data inputs, and its hypothesis outputs as reducible to its physical posits. Before "Building Phenomenological Bridges" there was: Orseau & Ring (2011); Orseau & Ring (2012).The post 2013 in Review: Friendly AI Research appeared first on Machine Intelligence Research Institute.

2013 in Review: Friendly AI Research


MIRI's February 2014 Newsletter

Dear friends,
See below for news on our new ebook, new research, new job openings, and Google's new AI ethics board.
Research Updates

New analyses: Robust Cooperation: A Case Study in Friendly AI Research and How Big is the Field of Artificial Intelligence?
New interviews: Emil Vassev on formal verification, Mike Frank on reversible computing, Ronald de Wolf on quantum computing, Gerwin Klein on formal methods, André Platzer on verifying cyber-physical systems.
Two MIRI talks from the AGI-11 academic conference have finally been posted (videos + slides + transcripts).

News Updates

MIRI has a new Careers page, currently with 4 job openings.
New ebook: Smarter Than Us: The Rise of Machine Intelligence.
Want to help MIRI by investing in XRP?
Two new posts in our 2013 in Review series, on outreach and on strategic and expository research.
Two new conversations about MIRI strategy, with Holden Karnofsky and with Jacob Steinhardt.
MIRI's experience with Google Adwords.

Other Updates

If you're in the UK, go see Martin Rees, Jaan Tallinn, and Huw Price at the Cambridge lecture "Existential Risk: Surviving the 21st Century" on Feb. 26th.
The big AI news last month was Google's ~$400M acquisition of DeepMind, and Google's rumored creation of an AI ethics board. The best coverage I saw was at Forbes: Inside Google's Mysterious Ethics Board. One of DeepMind's co-founders, Shane Legg, wrote his PhD thesis on machine superintelligence, takes AI risks quite seriously, and previously wrote about strategies for funding safe AGI. (Shane is far from the only safety-conscious person at DeepMind; he's just the one who has written the most about AGI safety.)
Blue Marble Space is running an essay contest on preparing for the distant future of civilization. $500 first-place prize.
Nick Bostrom's next book is Superintelligence: Paths, Dangers, Strategies, now available for pre-order from Amazon US and Amazon UK. I've read it three times already, and it is excellent. It's easily the most cohesive, comprehensive coverage of the subject so far.

As always, please don't hesitate to let us know if you have any questions or comments.

Best,
Luke Muehlhauser
Executive Director

The post MIRI's February 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s February 2014 Newsletter


New eBook: 'Smarter Than Us'

We are pleased to release a new ebook, commissioned by MIRI and written by Oxford University's Stuart Armstrong, and available in EPUB, MOBI, PDF, and from the Amazon and Apple ebook stores.
What happens when machines become smarter than humans? Forget lumbering Terminators. The power of an artificial intelligence (AI) comes from its intelligence, not physical strength and laser guns. Humans steer the future not because we're the strongest or the fastest but because we're the smartest. When machines become smarter than humans, we'll be handing them the steering wheel. What promises—and perils—will these powerful machines present? Stuart Armstrong's new book navigates these questions with clarity and wit.
Head over to intelligence.org/smarter-than-us/ to grab a copy.
The post New eBook: 'Smarter Than Us' appeared first on Machine Intelligence Research Institute.

New eBook: ‘Smarter Than Us’


André Platzer on Verifying Cyber-Physical Systems

André Platzer is an Assistant Professor of Computer Science at Carnegie Mellon University. He develops the logical foundations of cyber-physical systems to characterize their fundamental principles and to answer the question how we can trust a computer to control physical processes. He received an M.Sc. from the University of Karlsruhe (TH), Germany, in 2004 and a Ph.D. in Computer Science from the University of Oldenburg, Germany, in 2008, after which he joined CMU as an Assistant Professor.
Dr. Platzer received a number of awards for his research on logic of dynamical systems, cyber-physical systems, programming languages, and theorem proving, including an NSF CAREER Award and ACM Doctoral Dissertation Honoroable Mention Award. He was also named one of the Brilliant 10 Young Scientists by the Popular Science magazine and one of the AI's 10 to Watch by the IEEE Intelligent Systems magazine.

Highlights of Platzer's thoughts, from the interview below:

KeYmaera is a formal verification tool for cyber-physical systems. It has seen many real-world applications, e.g. verifying non-collision in the European Train Control System.
KeYmaera's capability comes from Platzer's "differential dynamic logic," which builds on earlier work in (e.g.) dynamic logic.
Contra Levitt, Platzer isn't convinced that autonomous robots will need to use decision theory, though they may.
Platzer contrasts formal verification ("did I build the system right?") with formal validation ("did I build the right system?"). One way to make progress on the latter is to develop what Rushby called "deep assumption tracking."

Luke Muehlhauser: One of your projects is KeYmaera, a tool for deductive formal verification of hybrid systems, including autonomous control systems for trains, planes, and robots. What counts as a "hybrid system," in this context? What commercial or government applications do you see as being within KeYmaera's plausible capabilities reach, but for which KeYmaera isn't yet being used?

André Platzer: KeYmaera is a tool for verifying hybrid systems. A hybrid system — or, more precisely, a hybrid dynamical system — is one in which both discrete dynamics and continuous dynamics interact. In other words, part of the system exhibits a discrete behavior, often caused by computation or communication, and other parts of the system exhibit a continuous behavior, such as physical motion in space or continuous chemical reactions. Systems in classical engineering have predominantly continuous dynamics while computer science classically explores only discrete dynamics, because, after all, computers compute one thing after another and are, thus, inherently discrete. Communication is a similarly discrete phenomenon where information is exchanged once at a certain point in time and then again later on at some other point in time. For example, over a wireless channel, communication does not literally happen continuously all the time, only every once in a while. However fast it might appear to the innocent bystander.
Hybrid systems now start with the realization that exciting things happen as soon as you make discrete dynamics and continuous dynamics interact. All of a sudden, your systems can, for example, use discrete computers and the power of advanced software algorithms to control and influence continuous physical processes and make them safer.
But we can no longer quite understand the discrete and continuous part of the system in isolation, because they interact. It would not be enough to worry just about whether the software could crash but also about whether it feeds the right control input into the physical system that it is trying to control and what effect those inputs will have on the physics. Likewise, it is not sufficient to understand the continuous dynamics alone, because its effect depends on the control input generated by the computer program.
This is where hybrid systems start and where KeYmaera comes in. Hybrid systems, which are a common core mathematical model behind cyber-physical systems (CPS), bring exciting new capabilities of how we can change the world for the better. But these capabilities come hand in hand with our responsibility of making sure that the systems are not going to let us down when we badly need them.
As Jeannette Wing has aptly phrased: "How can we provide people with cyber-physical systems they can bet their lives on?"
Application areas include aviation, automotive, railway, robotics, medical devices, physical or chemical process control, factory automation, and many more areas than I could list. Hybrid systems are a very general mathematical model which makes them interesting for many possible application areas. KeYmaera has indeed been used with industrial partners from the automotive industry, on mobile robotics applications of an industrial partner, in a study of a surgical robotics system by a government lab, and we are currently working with the Federal Aviation Authority on how to use KeYmaera for the next generation Airborne Collision Avoidance System ACAS X. Having said that, all those application areas bring up interesting challenges. Since most software in the wild still contains bugs, there is more potential and there remain more challenges in all the other application areas as well.

Luke: What role does differential dynamic logic play in KeYmaera? What was your inspiration for developing it?

André: Differential dynamic logic is the theoretical underpinning and mechanism behind KeYmaera. It is the principle that makes the verification power of KeYmaera work. Differential dynamic logic is the logic in which you can specify and verify properties of hybrid systems. The logic integrates all the relevant features of hybrid systems in a single language: differential equations for describing their continuous dynamics, difference equations / discrete assignments for discrete dynamics and conditional switching, nondeterministic choices, as well as repetition. With these fundamental dynamical principles, all the other relevant ones can be expressed.
I designed differential dynamic logic when I was starting my PhD. I was always very curious about how the core logical foundations of hybrid systems would look and simply could not wait to discover them. Especially since they beautifully brought together the discrete and continuous parts of mathematics, which are classically considered separately.
But then someone showed me a model of the European Train Control System (ETCS) with critical parameters (such as when to start braking) that were chosen more or less at random based on their performance in a few simulations. Suddenly I saw that there was also a real need in practice for the flexibility that differential dynamic logic provides. There simply had to be a better and more disciplined way of approaching the problem of how to design hybrid systems and safely choose their respective parameters. Any specific number for a crucial design parameter can obviously only be safe within a range of constraints on all the other parameters of the system. And, indeed, the chosen parameter value for the start braking point in the ETCS model turned out to cause a collision when just increasing the speed a little.
From this point on, I designed differential dynamic logic with a dual focus on both its core theory and its practical use in applications. This broader focus certainly made it more successful. Especially when practical applications raised interesting theoretical challenges and, conversely, questions considered for purely theoretical reasons later turned out to be critical in unsuspecting applications.
Differential dynamic logic is what we later used as the basis for implementing the theorem prover KeYmaera and has served us well since. Much has been written about differential dynamic logic since my first draft in 2005, including a book. But a good survey can be found in a LICS tutorial.

Luke: What previous work in logic and differential equations did you build on to create differential dynamic logic?

André: Differential dynamic logic fortunately had all of science to build on. An exhaustive list of all the areas of logic, mathematics, and computer science that its development and theory are drawing from would be longer than the whole interview. And the list with connections to control and engineering would be just as long. In fact, this is one of the reasons why research on hybrid systems is so exciting. It is certainly a way of being able to explore any of your favorite areas of mathematics while still having important applications that other people care about.
Fortunately, KeYmaera gives you all those powerful connections to other areas of science & engineering, but they are hidden behind the very elegant and intuitive language of differential dynamic logic. That means the user does not need to know all those areas of math and can play with the syntactical constructs of differential dynamic logic and its proof rules. Differential dynamic logic gives people a safe framework in which they can leverage and exploit all these areas without knowing all of them in depth. The computer can then check that the reasoning is combined in a correct way.
The most central previous work I built on is dynamic logic  originally due to Pratt and pioneered by many people including Harel, Meyer, Parikh, Kozen as well as on Sophus Lie's theory of continuous algebras and continuous groups. Both of those inventions are from before I was born, but have helped me a lot in designing differential dynamic logic.
Dynamic logic is a logic for conventional discrete programs that I hybridized by augmenting it with differential equations and generalizing it to hybrid systems with their interacting discrete and continuous dynamics. The original dynamic logic has also been implemented and used successfully in the prover KeY on which our prover KeYmaera is based. This heritage also explains the funny name KeYmaera, which makes independent sense, as KeYmaera is a homophone to Chimaera, the hybrid animal from ancient Greek mythology.
In retrospect, it is obvious that dynamic logic has really been longing for hybrid systems for three decades without anybody noticing. Dynamic logic was originally formulated for nondeterministic systems in which programs can have more than one possible effect. This is a fundamental characteristic but really slightly at odds with the behavior of conventional programs, which is mostly deterministic, i.e. each program statement will have exactly one effect. Well in hybrid systems, the story is different, because CPS implementations are never as deterministic as you might think they would run, and because the environment always has more than one possible future behavior that you need to consider. Dynamic logic and dynamical systems are simply made for each other.
Even more fundamental than dynamic logic itself are the general fundamental principles of logic. They led me to the realization that hybrid systems deserve a study of their logical characteristics. Hybrid systems used to be second class citizens because they were challenging and no one had created a logical foundation for them. Now that those proper logical foundations are in place, we see that hybrid systems and their programmatic expressions as hybrid programs are not just some peripheral artifact, but are actually a powerful construct in their own right. They possess the intrinsic characteristics of integrating interacting discrete and continuous behavior. The result of such an interaction may be a little bit like what happens when particles of fundamentally different origin collide in a high-energy physics collider experiment. Historically disjoint areas of mathematics & science suddenly being to interact. Really surprising interactions happen and scientific bridges are being built that unite discrete and continuous computation by way of the mediating nature of hybrid dynamics.
Another example illustrating the consequences of the impact of logic on hybrid systems is that proof theory led to a generalization of Lie's seminal results in ways that relate to Gentzen's seminal cut elimination in logic. Cuts are the logically fundamental mechanism behind lemmas in proofs and have been shown by Gentzen to be unnecessary. They are rather discrete, though, because cuts prove a (static) lemma and then use it. The counterpart for discrete cuts is called differential cuts which prove a property of a differential equation and then modify the system behavior to comply with this property. Differential cuts turn out to be fundamental and, for all I know, Sophus Lie does not seem to have been aware of the significance of such a logical mechanism for differential equations. Phenomena like this one demonstrate that amazing science can happen when logicians like Gerhard Gentzen interact with algebraists like Sophus Lie.
Yet, at the end of the day with all those mind-bending theoretical surprises, it is certainly also reassuring to observe their practical effect and relevance in verification and analysis results for CPS applications that have not been possible before. It is simply a nice mix to have exciting theory with useful applications at your fingertips every day.

Luke: Do you think the creators of dynamic logic, or the creators of other theoretical work that you built on to develop differential dynamic logic, thought that their work could be useful for program safety or verification one day? Or is this an instance of you plucking up theoretical results from unrelated disciplines and using them for your own purposes in software verification?

André: Well, these are all very smart people. I find it hard to imagine the pioneers of dynamic logic would not have thought of possible uses in program verification. Even if dynamic logic might have started from more theoretical / logical investigations, I am sure they were aware of this future. And I am not the first to use dynamic logic for verification either. But it is true that, unlike in the case of differential dynamic logic, the development of the theory, practice, and applications of dynamic logic was mostly led by different groups. Maybe it is just part of the natural appeal of hybrid systems that I can never resist to be as curious about their theory as I am about their applications.
Having said that, an excellent question is whether the inventors of dynamic logic have thought of hybrid systems. I, of course, don't know that. Dynamic logic predates the invention of hybrid systems by almost two decades. But hybrid systems have also almost always been around us. It's just that until the 1990s, nobody noticed that they are a fascinating mathematical phenomenon in their own right.
Concerning the other primary source of inspiration, the seminal work of Sophus Lie, he might not have thought of possible uses in the verification of hybrid systems, because Sophus Lie's results predate dynamic logic by a century. He invented what are now called Lie groups <PHONE>, which is long before computers were around. Yet, I do not want to leave the impression these were my only inspirations when developing differential dynamic logic and its theory and applications. There is significant influence also from ideas of Kurt Gödel, Gerhard Gentzen, Alfred Tarski, David Hilbert, Alonzo Church, Yiannis Moschovakis, Henri Poincaré, Joseph Doob, Eugene Dynkin, and many many others. I most certainly owe my results to what they have found out before.

Luke: Levitt (1999) argued that as intelligent systems become increasingly autonomous, and operate in increasingly non-deterministic, unknown environments, it will not be possible to model their environment and possible behaviors entirely with tools like control theory:
If robots are to put to more general uses, they will need to operate without human intervention, outdoors, on roads and in normal industrial and residential environments where unpredictable… events routinely occur. It will not be practical, or even safe, to halt robotic actions whenever the robot encounters an unexpected event or ambiguous [percept].
Currently, commercial robots determine their actions mostly by control-theoretic feedback. Control-theoretic algorithms require the possibilities of what can happen in the world be represented in models embodied in software programs that allow the robot to pre-determine an appropriate action response to any task-relevant occurrence of visual events. When robots are used in open, uncontrolled environments, it will not be possible to provide the robot with a priori models of all the objects and dynamical events that might occur.
In order to decide what actions to take in response to un-modeled, unexpected or ambiguously interpreted events in the world, robots will need to augment their processing beyond controlled feedback response, and engage in decision processes.
Thus, he argues, autonomous programs will eventually need to be decision-theoretic.
What do you think of that prediction? And in general, what tools and methods do you think will be most relevant for achieving high safety assurances for the highly autonomous systems of the future?

André: Levitt definitely has a point about the necessity of stronger safety arguments for robots, especially in light of their challenging and uncertain environments.
I would, however, not advise dismissing control theory entirely. I do not even see how decision theory would contradict control theory. Control theory is, without any doubt, contributing valuable technology and perspectives on cyber-physical systems, including robots. The spirit of cyber-physical systems and their mathematical model of hybrid systems is to learn from all the various disciplines involved and bring concepts together rather than segregating them.
While I sympathize with Levitt's conclusion about the possible use of decision-theoretical methods, I disagree with his argument. The reason why I do is a nice illustration of important principles behind cyber-physical systems. In the text you quoted and other places in his article, Levitt argues that control theory does not help because it requires models. He further argues that the solution would be to use decision theory, which, however, still needs models:
"For each step in a plan to accomplish a robot task, […] we can model the robot's immediate possible action choices as decisions. Then we can develop a priori sets of values for the possible task-relevant outcomes of robot actions."
In fact, this needs even more models: prior belief distribution models, models how perception updates beliefs, models of the cost of taking an action in a state.
It is inconclusive to argue that the cure to using models is to use more models, even if this argument occurs frequently. At the same time, it is hard to escape the Cartesian Demon without adopting at least some model of the world. As long as we acknowledge that our predictions are subject to some form of models, however hidden or data-driven they may be, there is, thus, nothing wrong in using models. And in fact, there is even a good reason to consider increasingly more flexible models.
Regardless, I agree with Levitt that robots live in an uncertain environment and, uncertainty, thus, needs to be included in models. Simple models are sufficient for understanding some phenomena in a system, while other aspects need more complicated models. But they are still models! Whether the right model is deterministic, probabilistic, nondeterministic, or adversarial depends on the exact purpose. In some situations, nondeterministic understandings of uncertainty are much better, in others, they have to be probabilistic, in yet others adversarial.
This is one of the central guiding principles behind multi-dynamical systems, which I define as systems that combine multiple dynamical aspects such as discrete, continuous, stochastic, nondeterministic, or adversarial dynamics. Multi-dynamical systems generalize hybrid systems and provide the dynamical features of cyber-physical systems that hybrid systems are missing. Differential dynamic logic generalizes very elegantly to dynamic logics of multi-dynamical systems.
This kind of flexibility is crucial for success in high assurance safety systems. It is simply a pity if all models, analysis results, and techniques need to be discarded entirely as soon as a dynamic aspect occurs that the techniques do not handle. If, instead, models and analysis techniques are as flexible as the family of differential dynamic logics, then there is no overwhelming cost associated with introducing or dropping nondeterminism or other dynamical aspects.
My colleagues Bruce Krogh, David Garlan and our students have argued for such flexibility of models as well, where the insight is that it is often more feasible to consider multiple heterogeneous models rather than a single all-embracing one.
Another frequent but faulty argument that Levitt expresses is
"In fact, we claim that if robot perceptions were certain, that we would have almost no uncertainty associated to robot behaviors."
The argument "if only we could sense, we could act" is a common fallacy. Its contrapositive is quite true: If we cannot sense, it is virtually impossible to control anything safely. Yet, even if the sensors happen to work perfectly, there are still numerous opportunities for subtle and blatant mistakes in controllers. And there are even more opportunities for mistakes coming from the uncertainty of the effects of actions that Levitt readily dismisses. For emphasis, I'll focus only on discussing controller mistakes that are independent of sensing and actuation uncertainty.
For example, our verification tool KeYmaera led Yanni Kouskoulas, one of my collaborators at the John's Hopkins University's Applied Physics Lab (JHU APL), to find and fix two very subtle flaws in the controllers of a surgical robotics system for skull-base surgery. These controllers had been developed according to the best practice state of the art in safety-critical systems and had been scrutinized by their developers. Yet, the developers had second thoughts and approached the JHU APL, who ultimately approached us. The fact that the bugs have only been found by formal verification in the end demonstrates that it is just extraordinarily difficult to get cyber-physical systems correct without the right analytic tools. But I am not surprised that it took sound verification tools to find the bugs. One was a timing bug caused by the inertia of the system. The other one had to do with very subtle acting and counteracting control depending on the precise geometrical relationships.
This is not an isolated case. We have observed similar benefits of using verification in our DARPA HACMS project. It was an interesting experience to observe the discrepancies that can arise when comparing formally verified parts of the robot to non-verified parts. That is always an eye-opening experience where all participants learn something important about the application domain.
Levitt underestimates substantial challenges:
"In summary we claim that valuing world states is computationally tractable in two senses. The first is that we can specify the values about world states a priori and use them to dynamically value actions during task execution. The second is that the number of such valuations is order linear in a relatively small number of abstract classes of world entities that concern a robotic task, such as roads, traffic signals, humans and animals."
But the complexity of decision making is quite a problem in practice. Even seemingly simple planning instances are NP-hard, which Bellman identified as the curse of dimensionality in the 1950s. Correct decisions that come half an hour too late are not particularly useful for rescuing a robot's safety mission. Real-time responses are critical. That's one of the reasons why it is crucial to have pre-verified safe responses as opposed to settling on unbounded online deliberation.
In contrast to what the above quote indicates, it has been an open challenge for decades to find out how to get the models right, get the values right, and make sure both translate into the right action to choose.
Consider ACAS X, the next generation Airborne Collision Avoidance System that the FAA tasked our collaborators at JHU APL and us to analyze with our verification techniques. Its design is built on decision theory in the form of Markov Decision Process optimization. On some level, it might sound trivial to identify that "no collisions ever" is the highest value in designing a collision avoidance system for aircraft. Yet, how exactly does this translate into math? What are the right secondary objectives that prevent unnecessary avoidance maneuvers? What is the right model to assume for an intruder aircraft? For pilot's responses? After many years of careful and highly skilled design of ACAS X, I have still seen an "optimal" interim version that is just not what anybody would want. And that, fortunately, will not fly.
Overall, I would argue that verification ultimately reduces correctness of cyber-physical systems to the question of whether we got the model right. It is a true challenge to identify the right model for a problem. But once we did, there are still numerous subtle ways of getting the answer wrong. These are the flaws that verification removes. Having said that, verification is also making progress on validating whether we got the right model in the first place.
For the record, I also strongly disagree with other claims Levitt made in that position paper. Especially:
"We claim that the core capability that makes ethical behaviors possible for robots is to understand the world external to the robot through the automated interpretation of sensory data. Robot perceptions have a fundamentally different relationship to the truth or falsity of statements about the world than is possessed by syntactic rules or similar representations."
This claim directly contradicts Alonzo Church's seminal proofs from 1936. And my primary line of research on logic and proofs for cyber-physical systems shows exactly how such relationships can be established and successfully exploited for verification. Thus, I understand Levitt's claims more as possible suggestions rather than conclusive arguments. That might be a logicians trait, though, to distinguish theorems from conjectures by whether or not they come with proof.

Luke: You wrote that "verification is also making progress on validating whether we got the right model in the first place." What did you have in mind, there? The thing that came to mind for me was Rushby (2013), but I don't know much about concrete ongoing progress on the problem of reducing (what he calls) "epistemic doubt."

André: John Rushby makes a great case for distinguishing logic doubt ("did we get the arguments right?") versus epistemic doubt ("did we get our knowledge of the world right?"). I agree with him that we can get logic doubt under control and make it a problem of the past. Yet, that is exactly what you need such strong sound reasoning frameworks for cyber-physical systems for, such as differential dynamic logic. The ultimate proof will simply have to take the dynamics of the system into account just as much as it takes its control into account.
Terminologically, I usually contrast formal verification ("did I build the system right?") with formal validation ("did I build the right system?"). That is primarily because verification versus validation has a widespread (but not unambiguous) compatible connotation in the field. And also because I want to emphasize that we should not settle only for arguments that Rushby aptly phrases as:
"Because an expert says so."
Yet, it doesn't matter what name we use, it's still our responsibility to make sure the system really works in the end.
Elements of formal validation are also used in our NSF Expedition project CMACS (Computational Modeling and Analysis for Complex Systems), led by my colleague Ed Clarke Among other things, we also probe models for plausibility by subjecting them to formal verification tests of whether or not they satisfy expected properties. These are not only the safety-critical properties in the end, but also rather mundane properties whose only purpose is to validate whether the model behaves as expected. Certainly a technique that has been successful also in formal methods in chip designs.
To follow up on Rushby's suggestions, I completely agree with his plea for what I would call deep assumption tracking. That is, making sure all assumptions and conclusions are kept track of meticulously. Again an important topic in verification with multiple heterogeneous models. It's a common reason for minor and major hiccups if someone get's the assumptions wrong. Or if no one knows about them, so that other people draw the wrong conclusions. That's also why inadequate modeling assumptions that are left implicit tend to have the most dire consequences on a system's behavior.
In our work on provably safe obstacle avoidance for autonomous robotic ground vehicles, for example, our models went by subsequently enriching the dynamic capabilities of the obstacles from static over dynamic all the way to dynamic obstacles in the presence of sensor and actuator and location uncertainty. Since we accompanied each of those models with a correctness proof in KeYmaera, we couldn't help but notice under which circumstance the system is safe and when it's not. In our models, for example, it was a simple change to go from "obstacles are stuck" to "obstacles can move wherever they want, nondeterministically". Since the proof won't work then, though, you immediately realize that you first need to know some bound on the obstacle's velocity. Dynamic obstacles coming at you faster than the speed of light are simply hard to defend against.
Later, during a dry-run of a different controller implementation built without verification, it was interesting to see the difference in action. The unverified robot controller sometimes collided with obstacles, which is exactly what it wasn't supposed to do. A corresponding (simulation) run of the verified system instead vetoed unsafe control actions long before the collision ever happened. The reason why was prototypical. Whoever designed the unverified controller that we had access to did the natural thing: the robot first considered all obstacles static and hoped for fast enough sensing. Well, that is what our first verified model had also assumed. But our KeYmaera proofs immediately went further and investigated the case of dynamic obstacles, which is why we recommended to consider the case of dynamic obstacles for the unverified controller, too.
The unverified robot still collided. This time, the reason was more subtle. In a nutshell, the implementations silently assumed that walls would not move while other agents could. That sounds reasonably justified by everyday's experience. The catch is that corners do move. Well, they don't really. But as soon as your verification model says that obstacles either move or they don't and that it's their choice (nondeterministically!) where they move to, then you cannot help but notice in your proof that the system is only safe if your controller correctly addresses the case where an obstacle suddenly comes racing round a corner. And indeed, when we asked, we got the confirmation that troublemaker cases in practice were partially occluded obstacles.
Now it might be easy to blame the developers of the original robot controllers for this oversight and say that they just haven't been paying enough attention. But that would be too easy. In fact, that would miss the point, entirely. The point is that robotics and any other kind of cyber-physical systems design is just darn challenging. These are very hard problems. Programming is incredibly difficult. And programming robots only makes things more difficult, because we've suddenly got to face the whole physical reality around us. There's more ways to get them wrong, quite subtly, than there are ways of getting them right. We desperately need analytic support that helps us getting the designs right so that we can bet our lives on them.
Having talked to a number of people in academia and industry later on, it was not apparent to most of them that some static parts of the ambient space would have to be considered as moving even if they themselves didn't. This is a mistake that verification prevented us from doing.
But guess how we found out. A month earlier, postdocs in my group tried our freshly verified controller on some examples and came back quite frustrated, because it was too conservative for their taste. It shied away from narrow doorways. I was probably baffled for a moment there myself. But then it dawned on us that our verified controller was right. If obstacles can move anywhere they please, they might as well move quickly from behind a corner where our robot could not see them yet. Notice that we never built this knowledge into our model nor our controller nor anything else. We simply started with the canonical model of a robot's motion on a plane with a moving obstacle. Then we discovered where our proofs led us to in designing a verifiably safe controller. They led us to a controller that stays clear of narrow doorways. But that is what the robot simply has to do to stay provably collision-free.
We later discussed our verified controller with our collaborators in industry. They not only confirmed that our controller works within the intended operational range and wasn't overly conservative at all. But they also simply loved the sudden new capability of playing with different instantiations of the parameters of our verified model to find out whether the design will work correctly efficiently within their intended operating regime. Without having to set up all those different scenarios and system configurations experimentally. And the verified controller automagically ends up doing the right thing. When cutting corners, it will slow down to avoid collisions. When moving with a bigger margin around a corner, it would happily speed up. We didn't build that in. It just followed from the safety constraints of our proofs.
Still, we might have made mistakes in our model of the motion principles of the world and our assumptions about the environment. That would have given us a true, undeniable proof, which just does not apply to reality, because its assumptions aren't met. Or we might have misformalized what safety property it is that we care about. And this is where formal validation comes in. Techniques for ensuring that the models fit to the relevant part of reality and that the formulas fit to the intent. A problem that we just made some promising progress on and where I'm sure that many more exciting things will still be found out about.
Formal validation is just as exciting as formal verification. And that is why I am developing the Logical Foundations of Cyber-Physical Systems, because we simply need such strong analytic foundations when software reaches out into our physical world. Yet, with those foundations, there are suddenly amazing opportunities for how we can change the world for the better by designing cyber-physical systems that help us in our daily lives in more ways than we ever dreamed off. After all, true cyber-physical systems combine cyber capabilities with physical capabilities to solve problems that neither part could solve alone.

Luke: Thanks, André!
The post André Platzer on Verifying Cyber-Physical Systems appeared first on Machine Intelligence Research Institute.


Gerwin Klein on Formal Methods

Gerwin Klein is a Senior Principal Researcher at NICTA, Australia's National Centre of Excellence for ICT Research, and Conjoint Associate Professor at the University of New South Wales in Sydney, Australia. He is leading NICTA's Formal Methods research discipline and was the leader of the L4.verified project that created the first machine-checked proof of functional correctness of a general-purpose microkernel in 2009. He joined NICTA in 2003 after receiving his PhD from Technische Universität München, Germany, where he formally proved type-safety of the Java Bytecode Verifier in the theorem prover Isabelle/HOL.
His research interests are in formal verification, programming languages, and low-level systems. Gerwin has won a number of awards together with his team, among them the 2011 MIT TR-10 award for the top ten emerging technologies world-wide, NICTA's Richard E. Newton impact award for the kernel verification work, the best paper award from SOSP'09 for the same, and an award for the best PhD thesis in Germany in 2003 for his work on bytecode verification. When he is not proving theorems and working on trustworthy software, he enjoys travelling and dabbling in martial arts and photography. Together with Tobias Nipkow he has just published an online draft of the text book Concrete Semantics that teaches programming language semantics using the Isabelle theorem prover.

Highlights of Klein's thoughts, from the interview below:

Verifying code not designed for verification is very difficult and costly. Such "post-mortem verification" has other disadvantages as well.
Program designers can use abstraction, modularity, and explicit architecture choices to help make complex systems transparent to humans.
There are two ways probability can play a role in verification: (1) direct probabilistic reasoning, in the logic or in a setting where the program itself is probabilistic, or (2) standard non-probabilistic reasoning paired with subjectively uncertain reasoning about what a guarantee means for the overall probability that the system will work as intended.
Truly autonomous systems seem to be a long ways off, but if you could make a system with truly unexpected behavior, then you couldn't make it safe.

Luke Muehlhauser: In your forthcoming paper "Comprehensive Formal Verification of an OS Microkernel," you and your co-authors describe the kernel design (seL4) you used to ensure its verification would be tractable. You also discuss the process of keeping the formal proof current "as the requirements, design and implementation of the system [changed] over almost a decade."
How "micro" is the seL4 microkernel? That is, what standard functionality does it provide, and what functionality (that is common in larger OS kernels) does it not provide?

Gerwin Klein: It is pretty micro: it is a true micro-kernel in the L4 tradition. This means it is very small compared to traditional monolithic OS kernels: Linux has on the order of a few million lines of code, seL4 has on the order of 10,000 lines of C. seL4 provides only the core mechanisms that you need to build an OS. These include threads and scheduling, virtual memory, interrupts and asynchronous signals, as well as synchronous message passing. The maybe most important feature of seL4 is its strong access control mechanism that can be used together with the rest to build isolated components with controlled communication between them. These components could be small isolated device drivers or an entire Linux guest OS, the developer can choose the granularity.
The philosophy is: if a mechanism strictly needs privileged hardware access to be implemented, it is in the kernel, but if it can be implemented using the other mechanisms as a user-level task, it is outside the kernel. A surprising amount of features can be outside: device drivers, for instance, graphics, files and file systems, network stacks, even disk paging is not provided by seL4, but can be implemented on top of it. What you (hopefully) get is that tricky complexity is concentrated in the kernel, and user-level OS components become more modular.

Luke: Is anyone likely to formally verify an OS microkernel of seL4's size, or any computer program of comparable complexity, if it wasn't designed from the ground up to be especially amenable to formal verification?

Gerwin: I wouldn't say it's impossible to verify code that wasn't designed for verification, but it is going to increase cost and effort significantly. So currently it's unlikely that someone would do a similar verification on something that wasn't designed for it. This could change in 5-10 more years as techniques become more powerful and effort decreases.
But even if it is possible, I don't think it is desirable. I've heard someone call this kind of process "post-mortem verification" and I agree with that general sentiment. Starting verification after everything else is finished is too late. If you find serious problems during the verification, what can you do about them if everything is set in stone already? Also, if you are serious about building a provably correct system or a high-assurance system, you should design and plan it as such. Interestingly, agile methods go into the same direction: they test and verify right from the get-go, even if they don't use formal methods for it.
If you can factor the cost of verification into your development process, your design choices will be more biased towards making it obvious why the system works correctly. In our experience, this doesn't just reduce the cost of verification later, but also leads to better and more elegant designs. Of course there is also tension between verification and factors such as performance, but it's not black and white: you can evaluate cost and make informed trade-offs, and sometimes there are solutions that satisfy both.

Luke: You mention that designing with verification in mind may nudge design choices toward "making it more obvious why the system works correctly" — an important issue I've sometimes called "transparency," as in "transparent to human engineers." I wrote about this in Transparency and Safety-Critical Systems.
One of the "open questions" I posed in that post was: "How does the transparency of a method change with scale?" Perhaps a small program written with verification in mind will tend to be more transparent than a Bayes net of similar complexity, but do you think this changes with scale? If we're able to scale up formal methods to verify programs 10,000x bigger than seL4, would it be more or less transparent to human engineers than a Bayes net of comparable complexity? At least we can query the Bayes net to ask "what it believes about X," whereas we can't necessarily do so with the formally verified system. (You might also think I'm framing these issues the wrong way!)

Gerwin: What you call transparency probably is correlated closely with complexity, and complexity tends to increase with scale. There are a number of ways for dealing with complexity in traditional system design, for instance abstraction, modularity, and architecture. All of these also help with formal verification and make it easier to scale. For example, abstraction hides implementation complexity behind an interface and makes it possible to talk about a part  of a system without knowing all of the system.
A great early example of the power of abstraction is the London underground tube map: it tells you which trains go where and connect with which other trains, and it gives you a rough geographical relation, but it doesn't tell you which bends the tracks make, where precisely each station is, or what is above ground between stations. You can confidently navigate a fairly complex train system as a passenger without knowing anything at all about the more detailed mechanics of how everything works and where precisely everything is.
In our seL4 verification, we initially couldn't make much use of abstraction, because in a microkernel everything is tightly interrelated. In larger systems this is less necessary, in part because something like a kernel is then available to provide modules and isolation.
In the longer run, we benefited from abstraction even in the seL4 verification. We first verified that the code correctly implements a higher-level specification, which is an abstraction of the real kernel code. Years later, we used that much simpler abstraction, and, without even looking at the complex C code, could prove very specific security properties about it, such as that seL4 respects integrity and confidentiality. Since we had first proved that this specification was a true abstraction of the code, and because the properties we were interested were preserved by this kind of abstraction, we had automatically gained a verification of the code as well. The win in effort was significant: the verification of integrity was 20 times less expensive than the original code verification.
In my opinion, making serious use of such principles is going to be the way to scale verification. It is already what systems engineers do for complex but critical systems. The way to use it for verification is to stick with it more closely and not to make convenience exceptions that break abstractions or circumvent the system architecture. In the end, everything that makes it easier for humans to think about a system, will help to verify it.

Luke: You wrote that "There are a number of ways for dealing with complexity in traditional system design, for instance abstraction, modularity, and architecture." You gave examples of verifying complex code using abstraction and modularity — what did you have in mind for using architecture to make verification of complex systems more tractable?

Gerwin: Yes, thanks for spotting that.
There are two main ways I was thinking of in which good architecture can help verification. The first is just the fact that an explicit architecture and design will tell you which modules or layers exist in the system and how they are connected to each other. Making this explicit will enable you to draw some first conclusions about the system without even needing to look at the code. For example, if two components are not connected to each other, and the underlying operating system enforces the architecture correctly, then these two components will not be able to influence each other's behaviour, and that means you can verify each one in isolation. You now have two simpler verification problems instead of one large one.
The second way comes into play when you build the critical property you are interested in verifying directly into the architecture from the start. For example, imagine a trusted data logging component in a critical device. The properties you might want from this component are: it only accesses its own log files, it only writes data and never reads, and no other component in the system can write to those log files. To verify this property in a monolithic system, you'd either have to prove of all components with file write access that they don't overwrite the log file, or you would have to prove correct the access control mechanism of a complex file system. Instead of doing all that work, you could put an additional small filter component between any writers and the file system: the filter will deny requests to write to the log file and allow the rest. This is easy to implement and easy to prove correct. You now have reduced a large, potentially infeasible verification task to a (hopefully) small runtime cost and an architecture property.
For any of this to work, you need to make sure that the real code in the end actually adheres to the architecture and that the communication and protection structure of the architecture is enforced by the underlying system. This is why our group is thinking about such security architecture patterns: a formally verified microkernel is the perfect building block for building high assurance systems in this way.

Luke: I'd like to get your perspective on something I previously asked Emil Vassev: To what degree can we demonstrate both deductive guarantees and probabilistic guarantees with today's formal verification methods? If we could generalize formal specification and verification procedures to allow both for deductive guarantees and probabilistic guarantees, perhaps we could verify larger, more complex, and more diverse programs.
Many of the deductive proofs for safety properties in today's formally verified systems are already "probabilistic" in the sense that the designers have some subjective uncertainty as to whether the formal specification accurately captures the intuitively desirable safety properties, and (less likely) whether there was an error in the proof somewhere.
But the question I'm trying to ask is about explicitly probabilistic guarantees within the proof itself. Example: the verifier could represent that it ran out of computation before it could prove or disprove whether the system had property X, but given its reasoning and search process thus far it's 80% confident that the system has property X, kind of like human researchers today can't prove whether P = NP or not, but they're (say) subjectively 80% confident that P ≠ NP. Though perhaps to do this, we'd first need a solution to the problem of how a formal system or logical calculus can have internally consistent probabilistic beliefs (and belief updates) about logical sentences (a la Demski 2012 or Hutter et al. 2012).

Gerwin: Emil gave a great answer to this question already, so I'll try to approach it from a slightly different angle. I see two ways in which probability can play a role in verification: one where you reason about probabilities directly, be it just in the logic or even in a setting where the program itself is probabilistic, or another one where you reason classically, but are trying to figure out what that means for the overall probability that your system will work as intended.
There are good examples for both of these methods, and both are useful to apply, even within the same project. As Emil pointed out, it is preferable to use full deduction and have the ability to draw definite conclusions. But when that is not available, you may be satisfied with at least a probabilistic answer. There are a number of probabilistic model checkers, for instance PRISM from Marta Kwiatkowska's group in Oxford, or the Probabilistic Consistency Engine (PCE) from SRI. These tools will let you attach probabilities to certain events in a system or attach probabilities to beliefs about the state of the world and reason from there. The Probabilistic Consistency Engine is very close to what you mentioned in the question: reconciling potentially conflicting probabilistic beliefs about the world and drawing probabilistic conclusions from those descriptions.
There are also programs and protocols that directly work with probability, and for instance virtually flip a coin to make certain decisions. The Firewire protocol for instance uses this mechanism, and despite its use of probability you can make definite claims about its behaviour. Carroll Morgan who is working with us at UNSW has been applying probabilistic reasoning about such programs to security questions and program construction. To give you a flavour of what is possible in this space: one of our PhD students, David Cock, has recently used Carroll's methods in Isabelle to reason about covert channels in microkernels and the maximum bandwidth at which they can leak information.
Finally, there is the question how uncertainty about requirements and uncertainty about the physical world fits formal verification. John Rushby from SRI has a great take on this question in his paper about possibly perfect systems Logic and Epistemology in Safety Cases. His point is that there are two different kinds of probability to consider: the probability that the system is correct, and the underlying uncertainty we have about the world. For example, a utopian fully verified system where we captured all requirements correctly and where we got everything perfect might still fail, if one of its physical components fails from wear and tear or overheating or because someone dropped it into a volcano.
Formal verification does nothing about the latter kind of probability, but it increases the probability that the system is perfect (even if it can never fully get there). We can use this concept to figure out where verification has the largest impact, and quantify how much impact it has. For example, it may not make much sense to formally verify a system where even the best possible solution could only be right about 50% of the time, because the world around it behaves too randomly. On the other hand, we can get a huge pay-off from formal verification in a setting where requirements and system can be captured precisely. Rushby's approach lets us quantify the difference.

Luke: Levitt (1999) argued that as intelligent systems become increasingly autonomous, and operate in increasingly non-deterministic, unknown environments, it will not be possible to model their environment and possible behaviors entirely with tools like control theory:
If robots are to put to more general uses, they will need to operate without human intervention, outdoors, on roads and in normal industrial and residential environments where unpredictable… events routinely occur. It will not be practical, or even safe, to halt robotic actions whenever the robot encounters an unexpected event or ambiguous [percept].
Currently, commercial robots determine their actions mostly by control-theoretic feedback. Control-theoretic algorithms require the possibilities of what can happen in the world be represented in models embodied in software programs that allow the robot to pre-determine an appropriate action response to any task-relevant occurrence of visual events. When robots are used in open, uncontrolled environments, it will not be possible to provide the robot with a priori models of all the objects and dynamical events that might occur.
In order to decide what actions to take in response to un-modeled, unexpected or ambiguously interpreted events in the world, robots will need to augment their processing beyond controlled feedback response, and engage in decision processes.
Thus, he argues, autonomous programs will eventually need to be decision-theoretic.
What do you think of that prediction? In general, what tools and methods do you think will be most relevant for achieving high safety assurances for the highly autonomous systems of the future?

Gerwin: There's a lot of work ahead of us in this area, and I don't think we have very good solutions yet. After all, we're still struggling to build simpler systems properly.
On the other hand, I agree that more autonomy seems to be the direction that systems are heading towards. In the DARPA HACMS program that Kathleen Fisher is managing we are investigating some approaches to high-assurance autonomous vehicles. One thing you can do for instance, is use a so-called simplex architecture. You use two controllers: one complex, autonomous controller and an additional, simpler safety controller that you can provide high assurance for. During normal operation, the complex autonomous controller runs, and provides convenient, even smart behaviour. The simple safety controller stays in the background and monitors operations. It only checks whether the vehicle is still within safe parameters. Should something unforeseen happen, such as a fault, a security attack, or simply a defect in the complex controller, the simple one takes over and puts the system in a safe mode. For autonomous air vehicles, this could be "fly home to base", or "land safely in a free spot". This solution doesn't work for everything, but it is a good first step.
Whether we will really start building systems that come up with their own unexpected behaviour, based on independent reasoning, or not, I can't really say. In current autonomous vehicles, like self-driving cars or proposals like delivery drones, this is certainly not yet the case. And I have a hard time imagining regulatory authorities agreeing to something like this (with good reasons). Instead of trying to predict everything, these systems currently are constructed to safely deal with classes of problems. For instance, you don't need to know precisely which kind of obstacles you will encounter for a self-driving car. You don't need to know if it is an animal, debris, or a small child. It's enough to see that something is in your way or is on a collision course and react to this better than humans do. And you err on the side of caution rather than take risks. To get an improvement over the current state of human drivers for instance, you don't need to build a system that is perfect. You only need to build one that drives a lot safer than humans. That is a much simpler problem.
Systems that do their own reasoning could still be made safe, if you can show that the reasoning is sound and you can explore the implication space of the situations you expect it to be in. The problem is that automated reasoning is hard — not just complicated to think about, but a theoretically hard problem where even simple problems need a lot of computational power. Building systems directly even for complex situations is currently still a lot easier than that, so I'd say that autonomous programs that do their own reasoning are still closer to science fiction than reality. Asimov played with this vision in his famous laws of robotics, which in his stories often lead to surprising, but perfectly valid (according to those laws) behaviours. In the end, if you really want to build a system that can have truly unexpected behaviour, then by definition you cannot verify that it is safe, because you just don't know what it will do. You'd have to deal with them more like you deal with people than you deal with machines.

Luke: What do you think are some promising "next steps" — say, over the next decade — for research on how to achieve high-assurance AI systems?

Gerwin: There's a lot of work ahead of us in this area, and I don't think we have very good solutions yet. After all, we're still struggling to build simpler systems properly.
I think research such as in the HACMS program is encouraging: high-assurance control and sensing algorithms, high-assurance operating systems, not just kernels, but including the wider system infrastructure, high-level design tools that allow architecture-driven approaches and analysis, and generally stronger verification tools will be the key to make such autonomous systems more reliable and safer.
If we can crack the next order of magnitude in system scale, things will get a lot more interesting. There are some good approaches for this developing: instead of trying to find proofs without context, we have had some initial success in synthesizing more code from specifications and generating the corresponding proofs automatically. If that approach works on a larger scale, it could make building high-assurance systems dramatically cheaper in the future.

Luke: Thanks, Gerwin!
The post Gerwin Klein on Formal Methods appeared first on Machine Intelligence Research Institute.

Gerwin Klein on Formal Methods


Conversation with Jacob Steinhardt about MIRI Strategy

On January 21st, 2014, MIRI met with Jacob Steinhardt to discuss MIRI strategy. Participants were:

Luke Muehlhauser (executive director at MIRI)
Jacob Steinhardt (grad student in computer science at Stanford)

We recorded and transcribed much of the conversation, and then edited and paraphrased the transcript for clarity, conciseness, and to protect the privacy of some content. The resulting edited transcript is available in full here (6 pages).
Topics discussed include research communication style, inferential distance, FAI research directions, and Knightian uncertainty.
The post Conversation with Jacob Steinhardt about MIRI Strategy appeared first on Machine Intelligence Research Institute.