Max Tegmark on the mathematical universe

Known as “Mad Max” for his unorthodox ideas and passion for adventure, his scientific interests range from precision cosmology to the ultimate nature of reality, all explored in his new popular book “Our Mathematical Universe“. He is an MIT physics professor with more than two hundred technical papers, 12 cited over 500 times, and has featured in dozens of science documentaries. His work with the SDSS collaboration on galaxy clustering shared the first prize in Science magazine’s “Breakthrough of the Year: 2003.”

Luke Muehlhauser: Your book opens with a concise argument against the absurdity heuristic — the rule of thumb which says “If a theory sounds absurd to my human psychology, it’s probably false.” You write:
Evolution endowed us with intuition only for those aspects of physics that had survival value for our distant ancestors, such as the parabolic orbits of flying rocks (explaining our penchant for baseball). A cavewoman thinking too hard about what matter is ultimately made of might fail to notice the tiger sneaking up behind and get cleaned right out of the gene pool. Darwin’s theory thus makes the testable prediction that whenever we use technology to glimpse reality beyond the human scale, our evolved intuition should break down. We’ve repeatedly tested this prediction, and the results overwhelmingly support Darwin. At high speeds, Einstein realized that time slows down, and curmudgeons on the Swedish Nobel committee found this so weird that they refused to give him the Nobel Prize for his relativity theory. At low temperatures, liquid helium can flow upward. At high temperatures, colliding particles change identity; to me, an electron colliding with a positron and turning into a Z-boson feels about as intuitive as two colliding cars turning into a cruise ship. On microscopic scales, particles schizophrenically appear in two places at once, leading to the quantum conundrums mentioned above. On astronomically large scales… weirdness strikes again: if you intuitively understand all aspects of black holes [then you] should immediately put down this book and publish your findings before someone scoops you on the Nobel Prize for quantum gravity… [also,] the leading theory for what happened [in the early universe] suggests that space isn’t merely really really big, but actually infinite, containing infinitely many exact copies of you, and even more near-copies living out every possible variant of your life in two different types of parallel universes.
Like much of modern physics, the hypotheses motivating MIRI’s work can easily run afoul of a reader’s own absurdity heuristic. What are your best tips for getting someone to give up the absurdity heuristic, and try to judge hypotheses via argument and evidence instead?

Max Tegmark: That’s a very important question: I think of the absurdity heuristic as a cognitive bias that’s not only devastating for any scientist hoping to make fundamental discoveries, but also dangerous for any sentient species hoping to avoid extinction. Although it appears daunting to get most people to drop this bias altogether, I think it’s easier if we focus on a specific example. For instance, whereas our instinctive fear of snakes is innate and evolved, our instinctive fear of guns (which the Incas lacked) is learned. Just as people learned to fear nuclear weapons through blockbuster horror movies such as “The Day After”, rational fear of unfriendly AI could undoubtedly be learned through a future horror movie that’s less unrealistic than Terminator III, backed up by a steady barrage of rational arguments from organizations such as MIRI.
In the mean time, I think a good strategy is to confront people with some incontrovertible fact that violates their absurdity heuristic and the whole notion that we’re devoting adequate resources and attention to existential risks. For example, I like to ask why more people have heard of Justin Bieber than of Vasili Arkhipov, even though it wasn’t Justin who singlehandedly prevented a Soviet nuclear attack during the Cuban Missile Crisis.

Luke: After reviewing mainstream contemporary physics, you begin to explain the “multiverse heirarchy” in chapter 6, which refers to four “levels” of multiverse we might inhabit. The “Level I multiverse,” as you call it, is an inescapable prediction of what is currently the most widely accepted theory of the early universe (eternal inflation): the universe is infinite in all directions, implying that there is an identical copy of me, who is also asking Max Tegmark questions via email, roughly 101029 meters from where I sit now.

To many readers that will sound absurd. But you write that it is “a prediction of eternal inflation which, as we’ve seen above, agrees with all current observational evidence and is implicitly used as the basis for most calculations and simulations presented at cosmology conferences.” Moreover, the Level I multiverse could still exist even if eternal inflation turns out to be false. All we need for the Level I multiverse is, you write:

Infinite space and matter: Early on, there was an infinite space filled with hot expanding plasma.
Random seeds: Early on, a mechanism operated such that any region could receive any possible seed fluctuations, seemingly at random.

And as you explain, we have pretty decent evidence for these two claims, independent of whether eternal inflation in particular happens to be correct.
Still, I’m curious what your colleagues in physics departments around the world think of this. If you had to guess, what proportion of them accept that the Level I multiverse is a straightforward prediction of eternal inflation? And roughly what proportion of them think eternal inflation, or some theory that assumes both “infinite space and matter” and also “random seeds”, will turn out to be correct?

Max: There’s definitely been an increased acceptance of these ideas, with the most vocal critics shifting from saying “this makes no sense and I hate it!” to “I hate it!”, tacitly acknowleding that it’s actually a scientifically legitimate possibility.
I haven’t seen any relevant poll, but my sense is that the proportion of physicists who think a Level I multiverse is likely depends strongly on their subfield of physics, with the proportion being highest among theoretical cosmologists and high-energy theorists, and lowest in very different areas where people don’t normally think much about these ideas and often feel that they sound too weird to be true.
You and your MIRI colleagues work very hard to be rational, so if you’re convinced that A implies B and A is true, then you’ll update your Bayesian prior to be convinced that B is also true. I suspect that many physicists are less rational than you: my guess is that many who are sympathetic toward inflation and learn that it generically implies eternal inflation and Level I don’t actually update their prior about Level I, but instead tell themselves that a multiverse feels unscientific, and therefore lose interest in spending more time thinking about consequences of inflation.

Luke: You go on to explain four levels of multiverse in total:

Level I multiverse: “Distant regions of space that are currently but not forever unobservable; they have the same effective laws of physics but may have different histories.” A straightforward prediction of eternal inflation, and many other possible theories of cosmological evolution.
Level II multiverse: “Distant regions of space that are forever unobservable because space between here and there keeps in inflating; they obey the same fundamental laws of physics, but their effective laws of physics may differ.” Also suggested by eternal inflation.
Level III multiverse: “Different parts of quantum Hilbert space.” If the wavefunction never collapses but is instead always governed by the Schrödinger equation, this implies the universe is constantly “splitting” into parallel universes.
Level IV multiverse: “All mathematical structures, corresponding to different fundamental laws of physics.”
Before we get to Level IV, I want to ask about a claim you make about Level III.

You write that “fledgling technologies such as quantum cryptography and quantum computing explicitly exploit the Level III multiverse and work only if the wavefunction doesn’t collapse.”
But I assume that collapse theorists don’t think their view will be falsified as soon as the first “true” quantum computer is (uncontroversially) built. What do they argue in response? Why do you think quantum computing can only work if the wavefunction doesn’t collapse?

Max: That’s a good question, because there’s lots of confusion in this area. What’s uncontroversial is that quantum computers will work it there’s no collapse, i.e., if the Schrödinger equation works with no exceptions (as long as engineering obstacles such as decoherence mitigation can be overcome). What’s controversial is what happens otherwise. There’s such a large zoo of non-Everett interpretations (13 by now) that you’ll have to ask their adherents what exactly they predict for quantum computers, for quantum systems containing humans, etc. – in my experience, different people claiming to subscribe to the same interpretation sometimes nonetheless disagree on specific predictions.
The quantum litmus test is to ask them the following question:
Alice is in an isolated lab in a spaceship and measures the spin of an electron that was in a superposition of “up” and “down”. According to Bob, who hasn’t yet observed the spaceship, is Alice’s brain in a superposition of perceiving that she’s measured “up” and that she’s measured “down”?
The Many-Worlds Interpretation says unabiguously “yes”, because that’s what the Schrödinger equation predicts. Some Copenhagen supporters would say “no”, on the grounds that Alice collapsed the wavefunction when she observed the electron: something truly random happened at that instant, and it’s now really just up or really just down. Others say “yes” and still others will give you less clear-cut answers. My point is that a theory refusing to make a clear prediction for whether the Schrödinger holds for arbitrary large and complicated systems is automatically refusing to make a prediction for whether an arbitrarily large and complicated quantum computers work or not, and is therefore not a complete theory. I’d also argue that any theory where the wavefunction never collapses is simply Everett disguised in unfamiliar language – as far as nature is concerned, it’s only the equations that matter.

Luke: In chapter 11 you write that:
Whereas most of my physics colleagues would say that our external physical reality is (at least approximately) described by mathematics, I’m arguing that it is mathematics (more specifically, a mathematical structure). In other words, I’m making a much stronger claim. Why? …If a future physics textbook contains the coveted Theory of Everything (ToE), then its equations are a complete description of the mathematical structure that is the external physical reality. I’m writing is rather than corresponds to here, because if two structures are equivalent, then there’s no meaningful sense in which they’re not one and the same…
Do you have physics colleagues who assume external reality exists and that it can be described by mathematics, but who don’t accept your Mathematical Universe Hypothesis? If so, what are their counter-arguments?

Max: Interestingly, I haven’t heard any clearly articulated counter-arguments from physics colleagues. Rather, it’s a bit like with the unfriendly AI X-risk  argument: the scientists I know who are unconvinced by the conclusion don’t take issue with specific logical steps in the argument, but lack sufficient interest in the question to have familiarized themselves with the argument.
If you want to classify people’s views, it boils down to two logically separate questions:

Is our external physical reality completely described by mathematics?
Can something be perfectly described by mathematics (having no properties except mathematical properties) but still not be a mathematical structure?

The people I’ve heard answer “no” to 1) tend to do so not based on evidence or a logical argument, but based on a preference for a non-mathematical free will, soul or deity. The people I’ve heard answer “no” to 2) often conflate the description with the described, within mathematics itself. This ties in with the important question about whether mathematics is invented or discovered – a famous controversy among mathematicians and philosophers.
Our language for describing the planet Neptune (which we obviously invent – we invented a different word for it in Swedish) is of course distinct from the planet itself, which we discovered. Analogously, we humans invent the language of mathematics (the symbols, our human names for the symbols, etc.), but it’s important not to confuse this language with the structures of mathematics that I focus on in the book. For example, any civilization interested in Platonic solids would discover that there are precisely 5 such structures (the tetrahedron, cube, octahedron, dodecahedron and icosahedron). Whereas they’re free to invent whatever names they want for them, they’re not free to invent a 6th one – it simply doesn’t exist. It’s in the same sense that the mathematical structures that are popular in modern physics are discovered rather than invented, from 3+1-dimensional pseudo-Riemannian manifolds to Hilbert spaces. The possibility that I explore in the book is that one of the structures of mathematics (which we can discover but not invent) corresponds to the physical world (which we also discover rather than invent).

Luke: In chapter 13 you turn your attention to the future of physics, and the future of humanity within the physical world. In particular, you talk a lot about risks of human extinction, aka “existential risks.”
To summarize: the bad news is that there are lots of ways for humans to go extinct. The good news is that very few extinction risks are remotely likely in the next, say, 150 years. To illustrate this point, you provide this graphic:

I like that graphic, and I think it’s basically right, except that:

I’d downplay nuclear war as a fully existential risk (see here),
I’d change “global pandemic” to “synthetic biology” to emphasize that it’s novel pathogens that might be capable of full-blown human extinction (rather than “mere” global catastrophe),
and I’d add molecular nanotechnology as a major existential threat for the next 150 years.

I suspect the folks at Cambridge University’s Centre for the Study of Existential Risk (CSER) would make the same adjustments, as they also seem to be focusing on risks from synthetic biology, molecular nanotechnology, and AGI.
Do you think you’d agree with those adjustments, or is your basic picture somewhat different from mine on those points?

Max: I agree that “synthetic biology” is a better phrase, especially since the global pandemics I had in mind when making that figure were mainly human-made. The forms of molecular nanotechnology that I suspect pose the greatest existential risks are those that transcend the boundaries with synthetic biology or AI (already covered).
I disagree with the argument that we’ve overestimated nuclear war as an existential risk. Of course an all-out nuclear war couldn’t kill all humans instantly by literally blowing us up. However, I find the supposedly reassuring arguments you cite unconvincing, and had a spirited debate about this with one of the authors last year. To qualify something as an existential risk, we don’t need to prove that it will extinguish humanity – we simply need to establish reasonable doubt of the assertion that it can’t.
If the initial blasts disable much of our infrastructure and then nuclear winter lowers the summer temperature by about 20°C (36°F) in most of North America, Europe and Asia (as per Robock) to cause catastrophic crop failures, it’s not hard to imagine scenarios of truly existential proportions. Modern human society is a notoriously complex and hard-to-model system, so the scenarios I’m most concerned about involve complex interplays between multiple effects. For example, infrastructure breakdown might make it difficult to control either starvation-induced pandemics or armed gangs who systematically sweep the planet for food, weapons, etc. with little regard for sustainability. Without any serious attempts to model such complications, I don’t find the cited estimates of available biomass etc. particularly reassuring.

Luke: As Bostrom (2013) notes, humanity seems to be investing much more effort into (e.g.) dung beetle research than we are investing in research on near-term existential risks and how we might mitigate them. From your perspective, what can we do to cause research grantmakers (e.g. the NSF) and researchers (e.g. economists) to direct more of their effort toward research into near-term existential risks, especially the risks from novel technologies like synthetic biology and AGI?

Max: We need to draw more attention to these risks, so that people start thinking of them as real threats rather than science fiction fantasies. Organizations such as MIRI are an invaluable step in this direction.
We should aim to make more opinion leaders understand xrisks and make more people who understand xrisks into opinion leaders.

Luke: Thanks, Max!
The post Max Tegmark on the mathematical universe appeared first on Machine Intelligence Research Institute.


MIRI’s March 2014 Newsletter

Research Updates

We recently hired four new researchers, including two new Friendly AI researchers. We announced this to our local supporters at the recent MIRI Expansion Party.
We’ve scheduled our next research workshop for May 2014.
Three new posts on naturalized induction: Bridge Collapse, Solomonoff Cartesianism, and The Problem with AIXI.
New analyses: The world’s distribution of computation and Is my view contrarian?
Many new expert interviews: Anders Sandberg on space colonization, Nik Weaver on paradoxes of rational agency, John Baez on research tactics, Bob Constable on correct-by-construction programming, Armando Tacchella on safety in future AI systems, David Cook on VV&A, John Ridgway on safety-critical systems, Toby Walsh on computational social choice, and Randall Larsen & Lynne Kidder on USA bio-response.

News Updates

New post for 2013 in Review on Friendly AI research.
A conversation with Holden Karnofsky about future-oriented philanthropy.
MIRI and Our Final Invention were featured on FoxTV.
BioTrove interviewed Louie Helm about the future of AI, and Big Picture Science interviewed Luke Muehlhauser about intelligence in animals and machines.
Castify has turned several of Yudkowsky’s Sequences into professionally-read podcasts: Map and Territory, How to Actually Change Your Mind, A Human’s Guide to Words, Mysterious Answers to Mysterious Questions, Reductionism, and Ethical Injunctions.

Other Updates

Video of the inaugural lectures of the Center for the Study of Existential Risk at Cambridge University.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI’s March 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s March 2014 Newsletter


Recent Hires at MIRI

MIRI is proud to announce several new team members (see our Team page for more details):
Benja Fallenstein attended four of MIRI’s past workshops, and has contributed to several novel results in Friendly AI theory, including Löbian cooperation, parametric polymorphism, and “Fallenstein’s monster.” Her research focus is Friendly AI theory.
Nate Soares worked through much of the MIRI’s courses list in time to attend MIRI’s December 2013 workshop, where he demonstrated his ability to contribute to the research program in a variety of ways, including writing. He and Fallenstein are currently collaborating on several papers in Friendly AI theory.
Robby Bensinger works part-time for MIRI, describing open problems in Friendly AI in collaboration with Eliezer Yudkowsky. His current project is to explain the open problem of naturalized induction.
Katja Grace has also been hired in a part-time role to study questions related to the forecasting part of MIRI’s research program. She previously researched and wrote Algorithmic Progress in Six Domains for MIRI.
MIRI continues to collaborate on a smaller scale with many other valued researchers, including Jonah Sinick, Vipul Naik, and our many research associates.
If you’re interested in joining our growing team, apply to attend a future MIRI research workshop. We’re also still looking to fill several non-researcher positions.
The post Recent Hires at MIRI appeared first on Machine Intelligence Research Institute.

Recent Hires at MIRI


Toby Walsh on computational social choice

Toby Walsh is a professor of artificial intelligence at NICTA and the University of New South Wales. He has served as Scientific Director of NICTA, Australia’s centre of excellence for ICT research. He has also held research positions in England, Scotland, Ireland, France, Italy, Sweden and Australia. He has been Editor-in-Chief of the Journal of Artificial Intelligence Research, and of AI Communications. He is Editor of the Handbook of Constraint Programming, and of the Handbook of Satisfiability.

Luke Muehlhauser: In Rossi et al. (2011), you and your co-authors quickly survey a variety of methods in computational social choice, including methods for preference aggregation, e.g. voting rules. In Narodytska et al. (2012), you and your co-authors examine the issue of combining voting rules to perform a run-off between the different winners of each voting rule. What do you think are some plausible practical applications of this work — either soon or after further theoretical development?

Toby Walsh: As humans, we’re all used to voting: voting for our politicians, or voting for where to go out. In the near future, we’ll hand over some of that responsibility to computational agents that will help organize our lives. Think Siri on steroids. In such situations, we often have many choices as there can be a combinatorial number of options. This means we need to consider computational questions: How do we get computer(s) to work with such rich decision spaces? How do we efficiently collect and represent users’ preferences?
I should note that computer systems are already voting. The SCATS system for controlling traffic lights has the controllers of different intersections vote for what should be the common cycle time for the lights. Similarly, the Space Shuttle had 5 control computers which voted on whose actions to follow.
Computational social choice is, however, more than just voting. It covers many other uses of preferences. Preferences are used to allocate scarce resources. I prefer, for example, a viewing slot on this expensive telescope when the moon is high from the horizon. Preferences are also used to allocate people to positions. I prefer, for example, to be matched to a hospital with a good pediatrics depts. Lloyd Shapley won the Nobel Prize in Economics recently for looking at such allocation problems. There are many appealing applications in areas like kidney transplant, and school choice.
One interesting thing we’ve learnt from machine learning is that you often make better decisions when you combine the opinions of several methods. It’s therefore likely that we’ll get better results by combining together voting methods. For this reason, we’ve been looking at how voting rules combine together.

Luke: Which methods from social computational choice might be relevant to a situation like the following, adapted from Bostrom (2009)?
Suppose you want to learn agent A’s preferences, and suppose that you begin with a set of mutually exclusive preference orderings, and that you assign each of these some probability of representing A’s preferences. Now imagine that each of these preference orderings gets to send some number of delegates to Parliament. The number of delegates each preference ordering gets to send is proportional to the probability of that preference ordering. Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting.

Toby: In computational social choice, we’d be interested in computational aspects of such voting situations. The first problem in this setting is that the number of preference orderings is exponential. So, is the number of delegates also exponential which will make it computationally difficult to deal with such a large number of delegates? Or is it polynomial in which case, we need to consider which polynomial subset and how “representative” this is of the whole? Having solved such problems, we’ll then consider computational questions like how do the delegates compute their (possibly strategic) votes. We know from fundamental axiomatic results in social choice that any reasonable voting system is likely to be manipulable and subject to strategic voting.

Luke: In some of your work (e.g. 2011a, 2011b), you explore the computational complexity of manipulation. Are there cases where an agent can efficiently engage in truthful voting, but can’t efficiently engage in various forms of manipulation, including strategic voting?

Toby: Well, hopefully an agent can always efficiently engage in truthful voting as we want participation. But surprisingly, even this can be sometime challenging. A nice example goes back to the Victorian mathematician and writer Charles Dodgson, or to go by his more familiar pen name, Lewis Carroll. He proposed an interesting voting rule, known as Dodgson’s method which elects any candidate that is preferred by a majority of voters to any other candidate when such a candidate exists. When such a candidate does not exist (for instance, when votes are split three ways), it elects the closest candidate in the sense that we have to make the fewest number of pairwise swaps in the votes to get to a candidate that is preferred by a majority. It’s worth noting that many popular rules, even plurality voting, do not necessarily elect a candidate that is pairwise preferred to all others by a majority when such a candidate exists. Anyway, back to Dodgson’s rule. Somewhat to the surprise of many, even working out who wins an election run under Dodgson’s rule is computationally intractable in the worst case (formally, it is NP-hard).
Fortunately, most voting rules are easy to compute, typically even linear time. On the other hand, there are many deep impossibility results showing that voters may profit from strategic voting and mis-reporting their preferences. An interesting escape from this problem was put forwards by Bartholdi, Tovey and Trick. Perhaps strategic voting is possible but too computationally difficult to actually work out?
Since they, I and others have explored which voting rules have this property. For example, we recently settled a long standing open question that Borda voting (which many people will have seen as a variant of Borda voting is used in the Eurovision song contest) is computationally difficult to manipulate. These are worst case results so we’ve also looked at how difficult strategic voting is to compute in practice (not just in the worst case).

Luke: What are some of the most interesting questions in computational social choice that you think have a decent chance of being essentially resolved in the next 20 years?

Toby: We’ll definitely have a good idea of the computational properties of the many different voting rules proposed throughout history, from ancient ones like Borda to recent ones like Schulz’e rule. And we’ll understand the computational resistance of these rules to manipulation not just in the worst case but in average and in practice on real world elections. More practically, we’ll also have a good understanding of computational aspects of other problems in social choice like fair division. For instance, we’ll have developed a rich range of mechanisms for fair division that cover the spectrum of cases (divisible/indivisible goods, centralized/decentralized, etc) and for which we understand their properties (fairness, efficiency, etc) and agents behaviour (e.g. how agents can compute a best response, Nash dynamics, etc).

Luke: Thanks, Toby!
The post Toby Walsh on computational social choice appeared first on Machine Intelligence Research Institute.


Randall Larsen and Lynne Kidder on USA bio-response

Colonel Randall Larsen, USAF (Ret), is the National Security Advisor at the UPMC Center for Health Security, and a Senior Fellow at the Homeland Security Policy Institute, George Washington University. He previously served as the Executive Director of the Commission on the Prevention of Weapons of Mass Destruction Proliferation and Terrorism (2009-2010); the Founding Director and CEO of the Bipartisan WMD Terrorism Research Center (2010-2012), the Founding Director of the ANSER Institute for Homeland Security (2000-2003), and the chairman of the Department of Military Strategy and Operations at the National War College (1998-2000).

Lynne Kidder is the former President of the Bipartisan WMD Terrorism Research Center (the WMD Center) and was the principal investigator for the Center’s Bio-Response Report Card.  She is currently a Boulder, CO-based consultant, a research affiliate with the University of Colorado’s Natural Hazards Center, and also serves as the co-chair of the Institute of Medicine’s Forum on Medical and Public Health Preparedness for Catastrophic Events. Her previous positions include Sr. Vice President at Business Executives for National Security, Senior Advisor to the Center for Excellence in Disaster Management and Humanitarian Assistance (US Pacific Command), and eight years as professional staff in the U.S. Senate.

Luke Muehlhauser: Your Bio-Response Report Card assesses the USA’s bio-response capabilities. Before we explore your findings, could you say a bit about how the report was produced, and what motivated its creation?

Randall Larsen: The 9/11 Commission recommended that a separate commission examine the terrorism threat from weapons of mass destruction (WMD). The bipartisan leadership in the Senate and House asked former US Senators Bob Graham (D-FL) and Jim Talent (R-MO) to head the Congressional Commission on the Prevention of Weapons of Mass Destruction Proliferation and Terrorism (WMD Commission). The WMD Commission completed its work in December 2008 and published a report, World at Risk. In March 2009, the bipartisan leadership of Congress asked Senators Graham and Talent to re-establish the Commission to continue its work and provide a report card on progress. This was the first Congressional Commission to be extended for a second year.
I became the Executive Director for the WMD Commission’s second year, and in January 2010, the Commission released a WMD Report Card assessing 37 aspects of the WMD threat. The grades ranged from A’s to F’s. The failing grade that received the most attention, both on Capitol Hill and in the press, was the F grade for “preparedness to respond to a biological attack.”
At the commissioners’ final meeting in December 2009, they encouraged Senators Graham and Talent to continue their work with a focus on the biological threat. To do so, a not-for-profit organization (501c3) was created in March 2010, The Bipartisan WMD Terrorism Research Center (WMD Center). Senators Graham and Talented agreed to serve on the board of advisors, I became the CEO, and recruited Lynne Kidder to serve as the President.
Launching the project was a bit of a challenge, since many of the traditional national security organizations that support such work were solely focused on the nuclear threat—a reflection of the Congressional perspective. The legislation that created the WMD Commission had not contained the words bioterrorism or biology—ironic since World at Risk clearly identified bioterrorism as the most likely WMD threat.
We began work on the Bio-Response Report Card in January 2011 by recruiting a world-class team of senior advisors. They included a former Deputy Administrator of the Food and Drug Administration, a former Special Assistant to the President for Biodefense, the Director of Disaster Response at the American Medical Association, the VP and Director of RAND Health, the Founding President of the Sabin Vaccine Institute, and experts in the fields of public health, emergency medicine, and environmental remediation.
The Board of Advisors helped inform methodology of the project, helped define the categories of bio-response, and then proposed metrics in the form of questions, by which to assess capabilities in each category.

Luke: In the Methodology section of your report (p. 20), you explain that the project’s “Board of Advisors… informed project methodology, the categories of bio-response, and then proposed metrics by which to assess capabilities in each category,” and that you “enlisted a separate group of diverse subject-matter experts to perform much of the research and analysis required to answer these questions.” It looks like a challenging project, requiring hundreds or thousands of hours of labor from knowledgeable experts. It also looks as though, lacking an easily-imitable template that would satisfy the goals of the project, you had to come up with much of the methodology yourself. Could you give more detail as to the process that generated the report? A chronological account might be the easiest to remember and share.

Lynne Kidder: Yes, this was a very labor-intensive project. The biggest challenge was that no one had ever attempted a national-level, end-to-end assessment of bio-response capabilities ranging from initial detection through environmental cleanup. No federal department or agency has the ability to do such a study without getting considerable pushback for grading the work of many other organizations with related missions. This is why the study needed to be accomplished outside of government—preferably by a not-for-profit.
Additionally, we had to work with our Board of Advisors to develop a “scale of events”—none existed. This fact had caused great confusion within the national security and biosecurity communities. When some people talked of a biological attack, they had in mind an event similar to the anthrax envelopes of October 2001 (22 infected and 5 deaths), while others were referring to a catastrophic event that would threaten the lives of hundreds of thousands. Working with our Board of Advisors, we identified six scenarios (small-scale non-contagious, small-scale contagious, large-scale non-contagious, large-scale contagious, large-scale drug-resistant, and a global crisis). See page 21 of the report card for specific examples and details of each scenario.
We quoted George E. P. Box, “Essentially, all models are wrong, but some are useful.” Our model was not perfect, but we found it shocking that the federal government had not developed such a model for analysis, and we were convinced ours was useful for our study and as a standard for others to use.
The second step for our Board of Advisors was to complete our framework for analysis with a matrix. The vertical axis listed the eight (generally agreed upon) mission areas: Detection & Diagnosis, Attribution, Communication, Medical Countermeasure Availability, Medical Countermeasure Development & Approval Process, Medical Countermeasure Dispensing, Medical Management, and Environmental Cleanup. The horizontal axis listed the six scales of attack. (See page 9.)
We then discovered there were no agreed upon standards of performance for the eight mission areas, so we also had to develop them. We first defined each mission area, and then provided standards of performance–what we called, “Fundamental Expectations”. We concluded that the American public and their elected representatives would have these fundamental expectations based on the tens of billions of dollars that have been spent on biodefense since 2001. For instance, in the mission area of Medical Countermeasure Distribution & Dispensing, we listed these fundamental expectations:

Appropriate types of medical countermeasure stockpiles, strategically located, subject to rigorous security and environmental controls, with schedules of resupply, rotation, and shelf-life extension
Distribution and dispensing mechanisms that are timely, efficient and deliver medical countermeasures to the point of need
Redundant and community-based dispensing strategies developed to address specific population needs (age distribution, at risk populations, logistics, etc.)
Supporting communications strategies that are multi-lingual, multi-cultural, and multi-channel
A trained and knowledgeable workforce (professional and/or volunteer) with the skill set, willingness, and necessary preparation to participate in mass dispensing activities

The final step for our Board of Advisors was to develop a set of questions for each mission area—questions that would be provided to the eight teams of subject-matter-experts to evaluate the nation’s capability to meet the Fundamental Expectations. In the mission area of Medical Countermeasure Distribution & Dispensing the Board of Advisors provided six questions to the subject-matter-experts. Here is an example of one of the questions:
Can medical countermeasures in the Strategic National Stockpile be dispensed to affected populations within 48 hours (as specified in HSPD-21)?
Note: This is one of the very few areas where there actually was an agreed upon standard of performance.
Mission area definitions, scale of attacks, fundamental expectations, and questions were provided to eight teams of subject-matter-experts (SMEs). The SMEs provided detailed answers to the staff of the WMD Center. Senators Bob Graham and Jim Talent, Randy and I, plus two consultants (Dave McIntyre, PhD and Sara Rubin, MPH) prepared the report card—using the inputs from the Board of Advisors and SMEs—and assigned grades.

Luke: What process did you use to find and select the SMEs? Roughly how many ended up contributing, in how many different teams?

Randall: I have worked in the biodefense community since 1994, and Lynne Kidder had more than a decade’s experience –plus she was serving (and still is) as the Co-Chair of the Institute of Medicine’s Forum on Medical and Public Health Preparedness for Catastrophic Events. Between us, we knew most of the leading biodefense experts, in both the public and private sectors.
Our Board of Advisors were nationally-recognized senior leaders in the field. Their names were listed in our report, but they were only involved in setting up the study. This added credibility to our study, but since they were not involved in the assessment and grading, it protected these senior leaders from the highly-politicized environment in the nation’s capitol.
On the other hand, the SMEs we chose to answer the questions, and provide suggested grades, remained anonymous—so that any criticism of grades could only be directed at Graham/Talent/Larsen/Kidder.
We had a “team” of SMEs assigned to each of the eight mission areas, but in fact they weren’t real teams. Each individual worked independently—unaware of who else was answering the same questions. This allowed us to identify those areas in which there was general agreement, and more importantly, areas in which there was disagreement. For those mission areas that had widely different assessments (answers to our questions) we would follow up with the SMEs, and in some cases, recruited additional SMEs. For one mission area, we ended up with five SMEs.
Surprisingly (at least to us) there were few areas of major disagreement. When there was, it was often about interpretation of specific policies and guidelines provided by the federal government. For instance, was the “48-hour standard” (for getting antibiotics to affected populations) established in the Bush Administration, still the standard for the Obama Administration?
All of the Board of Advisors were paid honoraria, and the SMEs were paid consulting fees. We also reached out to other SMEs who agreed to participate pro bono–including government employees, industry representatives, and several from the academic/think tank community.

Luke: Are you able to reveal the total cost of preparing the report, or a ballpark figure? How was funding acquired?

Lynne: Our budget for the 10-month project was $500,000. Smith Richardson Foundation contributed $200,000 and the Skoll Global Threats Fund contributed $300,000.

Luke: In January 2010, a research associate at the James Martin Center for Nonproliferation Studies (CNS) published a somewhat-critical review of the earlier WMD report card by Bob Graham and Jim Talent, focusing on some of its claims about biosecurity. Were you aware of that article while preparing your followup, the Bio-Response Report Card (published Oct. 2011)? If so, did it influence your work on the Bio-Response Report Card? Do you have any general comments on the article?

Randall: We certainly agreed with the final sentence in the article, “Accordingly, their report card should be read and considered with a critical eye.” All reports, including the report written by Mr. Kirk Bansak, a research associate at CNS, should be carefully and thoughtfully read and considered.
Unfortunately, Mr. Bansak appears to have misunderstood the WMD Commission’s grading system. As described on page 5 of the WMD Commission’s Report Card, “This report card uses letter grades to assess the U.S. Government’s progress in implementing the Commission’s recommendations.”
These were not grades on the overall status of preparedness and nonproliferation efforts, rather grades on specific responses to the recommendations in World at Risk—the tasking given to the Commission by the bipartisan leadership in Congress when the commission was extended for a second year.
For instance: Recommendation 1-2 in World at Risk
“Develop a national strategy for advancing bioforensic capabilities.”
The report card gave this an A, commenting, “An Interagency Bioforensics Strategy has been finalized and approved by the U.S. Office of Science and Technology Policy and exceeds the criteria stated in the Commission’s recommendations.”
This was clearly not an A for bioforensics writ large, but specifically a grade for response to a World at Risk recommendation—“develop a national strategy.”
An F grade was given for Recommendation 1-5, “Enhance the nation’s capabilities for rapid response to prevent biological attacks from inflicting mass casualties.” Mr. Bansak states, “Graham and Talent, who also include logistical strategy and planning in their list of biothreat preparedness needs, also appear to overlook President Obama’s Executive Order assigning the U.S. Postal Service responsibility for dispensing medical countermeasures in the event of a biological attack…”
This was hardly overlooked. At the Commission meeting when grades were debated and assigned, top officials from the White House briefed the commissioners on a wide variety of WMD issues, and provided copies of the soon-to-be-released Executive Order regarding the U.S. Postal Service. The commissioners were not impressed with this two-page document.
As the Executive Director of the WMD Commission, I did not have a vote for the assignment of grades. I just counted the votes. In many areas, there was considerable discussion and debate amongst the commissioners—lengthy debates whether a particular area should receive a B- or C+, or a C- or D, and in one case, an A or A+. However, when we got to the issue of preparedness to respond to a biological attack (Recommendation 1 in World at Risk) the discussion was short and the grade unanimous—F.
Regarding the “too generous grade” for actions on the Biological Weapons Convention, we can only state, once again, Mr. Bansak appears to have misunderstood the grading criteria. Recommendation 2-4 in World at Risk stated, “Propose an action plan for achieving universal adherence to the Biological Weapons Convention.” The report card grade was a B+, based on the release of the National Strategy for Countering Biological Threats by the National Security Council in December 2009. A comment was also offered on what actions the Administration would need to take to raise the grade to an A.
Mr. Bansak comments on the B+ grade with, “This plaudit, while not unwarranted, seems inappropriate for this section because it overlooks the failing of the BWC…” Once again, we can only say, this was not a grade for the BWC. It was a grade for the government’s response to a specific recommendation in World at Risk.
It appears that the primary objection from Mr. Bansak and CNS is on the issue of verification—discussed at length in the CNS report. CNS is a strong supporter of improved verification, however, the commissioners clearly agreed with the decisions of both the Bush and Obama Administrations that “bad verification is worse than no verification.”
The CNS assessment of the WMD Commission’s Report Card of January 2010 did not influence the WMD Center’s Report Card of October 2011. The primary purpose of the second report card was to move from a single letter grade of F (for the federal government’s response to a recommendation in World at Risk), to a much more detailed strategic analysis of this country’s capabilities to respond to a biological attack.

Luke: If a philanthropist granted your WMD Center $10 million to make research progress on the issue of national biodefense, and you were available full-time to lead such research efforts over the next five years, which project(s) might you execute? In other words, what do you see as the most urgent and desirable “next steps” for research into national biodefense?

Lynne: One top priority would be medical countermeasures (MCMs): developing new models where the public and private sectors can work in close cooperation to develop new MCMs for the Strategic National Stockpile (SNS); developing the capability for rapid development of MCMs in response to new bio-threats (both man-made and naturally-occurring); and the greatest MCM challenge, finding the means to rapidly gain FDA approval for use of new MCMs.
Another priority would be to promote best practices in states and local communities for biodefense preparedness. The majority of our work at the WMD Commission and WMD Center has been focused on federal programs. Senator Bob Graham, the former two-term governor of Florida, has encouraged looking at these issues from a local perspective—the frontlines of biodefense. There have been numerous private-public partnerships created at the state and local level to strengthen disaster preparedness and response. Unfortunately, the lessons learned and best practices from these pilot projects have not yet been fully exploited for wide spread use in cities and counties across the nation.

Randall: One specific challenge that will require effective private-public collaboration is the dispensing of medical countermeasures. The federal government has continually received high marks for maintaining the SNS and having the means to rapidly distribute the MCMs to cities and counties in a crisis. However, the problem is getting these MCMs into the hands (and mouths) of the citizens in these communities. If a bio-attack occurred in most American cities today, the feds would get the needed MCMs to the local airport, but there is serious doubt they would get to the residents in time to make a difference. The current situation is like having a fiber-optic cable with 1 gigabyte capability running down your street, but no link to your house.
Another important prospective project is executive education. Today’s leaders in Congress and the Administration do not fully appreciate the 21st century threat of bioterrorism. The biotech revolution has democratized the potential for creating biological weapons. The capability for sophisticated bio-attacks, that could threaten the lives of hundreds of thousands, was once limited to superpowers. Much has changed since the late 1960s. Today, these same (nuclear-equivalent) weapons, can be produced with equipment purchased on the internet and from pathogens readily-available in nature. Until senior leaders in both the public and private sectors fully understand this threat, it is unlikely they will give sufficient priority to developing the required defenses.

Luke: Thank you both!
The post Randall Larsen and Lynne Kidder on USA bio-response appeared first on Machine Intelligence Research Institute.


John Ridgway on safety-critical systems

John Ridgway studied physics at the University of Newcastle Upon Tyne and Sussex University before embarking upon a career in software engineering. As part of that career he worked for 28 years in the field of Intelligent Transport Systems (ITS), undertaking software quality management and systems safety engineering roles on behalf of his employer, Serco Transportation Systems. In particular, John provided design assurance for Serco’s development of the Stockholm Ring Road Central Technical System (CTS) for the Swedish National Roads Administration (SNRA), safety analysis and safety case development for Serco’s M42 Active Traffic Management (ATM) Computer Control System for the UK Highways Agency (HA), and safety analysis for the National Traffic Control Centre (NTCC) for the HA.
John is a regular contributor to the Safety Critical Systems Club (SCSC) Newsletter, in which he encourages fellow practitioners to share his interest in the deeper issues associated with the conceptual framework encapsulated by the terms ‘uncertainty’, ‘chance’ and ‘risk’. Although now retired, John recently received the honour of providing the after-banquet speech for the SCSC 2014 Annual Symposium.

Luke Muehlhauser: What is the nature of your expertise and interest in safety engineering?

John Ridgway: I am not an expert and I would not wish to pass myself off as one. I am, instead, a humble practitioner, and a retired one at that. Having been educated as a physicist, I started my career as a software engineer, rising eventually to a senior position within Serco Transportation Systems, UK, in which I was responsible for ensuring the establishment and implementation of processes designed to foster and demonstrate the integrity of computerised systems. The systems concerned (road traffic management systems) were not, initially, considered to be safety-related, and so lack of integrity in the delivered product was held to have little more than a commercial or political significance. However, following a change of safety policy within the procurement departments of the UK Highways Agency, I recognised that a change of culture would be required within my organisation, if it were to continue as an approved supplier.
If there is any legitimacy in my contributing to this forum, it is this: Even before safety had become an issue, I had always felt that the average practitioner’s track record in the management of risk would benefit greatly from taking a closer interest in (what some may deem to be) philosophical issues. Indeed, over the years, I became convinced that many of the factors that have hampered software engineering’s development into a mature engineering discipline (let’s say on a par with civil or mechanical engineering) have at their root, a failure to openly address such issues. I believe the same could also be said with regard to functional safety engineering. The heart of the problem lies in the conceptual framework encapsulated by the terms ‘uncertainty’, ‘chance’ and ‘risk’, all of which appear to be treated by practitioners as intuitive when, in fact, none of them are. This is not an academic concern, since failure to properly apprehend the deeper significance of this conceptual framework can, and does, lead practitioners towards errors of judgement. If I were to add to this the accusation that practitioners habitually fail to appreciate the extent to which their rationality is undermined by cognitive biases, then I feel there is more than enough justification for insisting that they pay more attention to what is going in the world of academia and research organisations, particularly in the fields of cognitive science, decision theory and, indeed, neuroscience. This, at least, became my working precept.

Luke: What do you think was the origin of your concern for the relevance of philosophy and cognitive science to safety engineering? For example did you study philosophy, or Kahneman and Tversky, in university?

John: As a physics student I was offered little opportunity (and to be honest had little ambition) to pursue an education in philosophy. Of course it is true that quantum mechanics made a mockery of my cherished belief in the existence of unobserved, objective reality, but I was very much in the ‘I know it makes no sense whatsoever but just shut up and carry on calculating’ school of practice. In fact, it wasn’t until I had become a jobbing software engineer, investigating the use of software metrics to predict development timescales, that philosophical issues started to take on an occupational importance.
The practice within my department had been to use single-point estimations for the duration of tasks, then feed these figures and the task dependencies into a Gantt chart and simply use the chart to read off the predicted project end date. Sometimes the results were very impressive: predicted project end dates often proved to be in the right decade! Then one bright spark said, ‘You’re living in the dark ages. What you need are three-point estimations for the duration of each task (i.e. estimate a most likely duration together with an upper and lower band). You then use Monte Carlo Simulation to create a project outturn curve; a curve which indicates a statistical spread of possible project durations. Everyone does this nowadays. And it just so happens I have an expensive risk management tool to do all the nasty calculations for you’. Imagine his look of disgust when I told him that I still thought the old way was better!
Was he talking to a smug idiot? I like to think not. The basis of my objection was this: As a physicist, I was well aware of the benefits of using Monte Carlo Simulation in areas such as solid state and nuclear physics. Here it is used to determine the macroscopic behaviour of a physical system, where probability distribution curves can be used to model stochastic variability of the micro behaviour of the system under study. Now, however, I was being invited to use Monte Carlo methods to draw conclusions based upon the averaging of various levels of posited (and decidedly non-stochastic) ignorance regarding the future. In such circumstances, no one could provide me with a convincing argument for deciding the most appropriate form for the probability distribution curve upon which the simulation would be based. In fact, I was told this choice didn’t matter, though quite clearly it did. If, as seemed likely, a realistic distribution curve would have a fat tail, the results would be hugely influenced by the choice of curve. Furthermore, the extra two estimates (i.e. the upper and lower bands for task duration) were supposed to represent a level of uncertainty, but the uncertainty behind their selection was at least of the same order of magnitude as the uncertainty these bounds were supposed to represent. In other words, one could not be at all certain how uncertain one was! It occurred to me that no real information was being added to the risk model by using these three-point estimates, and so no amount of Monte Carlo Simulation would help matters.
This led me to philosophical musing about the nature of uncertainty and the subtleties of its relationship with risk. These musings had a potentially practical value because I thought that a lot of people were spending a lot of time and money using inappropriate techniques to give themselves false confidence in the reliability of their predictions. Unfortunately, none of my colleagues seemed to share my concern and all I could do to try and persuade them was to wave my hands whilst speaking vaguely about second order uncertainty, model entropy and the important distinction between variability and incertitude. So I decided there was a gap in my education that needed filling.
As it happened, my investigations soon led me to the work of Professor Norman Fenton of Queen Mary University London, and I familiarised myself with concepts such as subjective probability and epistemic versus aleatoric uncertainty. Furthermore, once subjectivity had been placed centre stage, the relevance of cognitive sciences loomed large and although I can’t claim to have studied Tversky and Kahneman, I became familiar with ideas associated with decision theory that owe existence to their work.
Suddenly, my career seemed so much more interesting. And once I moved on to safety engineering, the same issues cropped up again in the form of over-engineered fault tree diagrams replete with probability values determined by ‘expert’ opinion. Now it seemed all the more important that practitioners should think more deeply about the philosophical and psychological basis for their confident proclamations on risk.

Luke: In many of your articles for the Safety-Critical Systems Club (SCSC) newsletter, you briefly discuss issues in philosophy and cognitive science and their relevance to safety-critical systems (e.g. 1, 2, 3, 4, 5, 6). During your time working on safety engineering projects, how interested did your colleagues seem to be in such discussions? Were many of them deeply familiar with the issues already? Does philosophy of probability and risk, and cognitive science of (e.g.) heuristics and biases, seem to be part of the standard training for those working safety engineering — at least, among the people you encountered?

John: Perhaps my experience was atypical, but the sad fact is that I found it extremely difficult to persuade any of my colleagues to share an interest in such matters, and I found this doubly frustrating. Firstly, I thought it to be a missed opportunity on my colleagues’ part, as I felt certain that application of the ideas would be to their professional advantage. Their diffidence was to a certain extent understandable, however, since there was nothing in the occupational training provided for them that hinted at the importance of philosophy or psychology. However, what really frustrated me was the fact that no one appeared to be at all excited by the prospect of introducing these subjects into the workplace. How could that be? How could my colleagues fail to be anything other than utterly fascinated? In fact, their lack of interest seemed to me to represent nothing less than a wanton refusal to enjoy their job!
The key to the problem lay, of course, with the training provided by my employer, and it didn’t help that the internal department that provided such training glorified under the title of ‘The Best Practice Centre’. Clearly, anything that I might say that differed from the company endorsed view was, by definition, less then best! And I soon found that berating the centre’s risk management course for failing to explore the concept of uncertainty was, if anything, counter-productive. Upon reflection, I think that some of these frustrations led me to seek an alternative forum in which I could express my thinking. Publishing articles for the Safety Critical Systems Club newsletter provided such an outlet.

Luke: You say that you “felt certain that application of the ideas would be to their professional advantage.” Can you give me some reasons and/or example for why you felt certain of that?

John: I think that my concerns were the product of working within a profession that appears to see the world rather too much in frequentist terms, in which the assumption of aleatoric uncertainty would be valid. In reality, it is increasingly the case that the risks a systems safety engineer has to analyse are predicated predominantly upon epistemic uncertainty. I cite, in particular, safety risks associated with complex software-based systems, adaptive systems and so-called Systems of Systems (SoS), or indeed any system that interacts with its environment in a novel or inherently unpredictable manner. Whilst it is true that analysing stochastic failure of physical components may play a significant role in predicting system failure, the probabilistic techniques involved in such analysis simply cannot address epistemic concerns, i.e. where the parameters of any posited probability distribution curve may be a matter for pure speculation. (I am aware that Monte Carlo simulation is sometimes used to probabilistically model the parametric uncertainty in probabilistic models, but this strikes me as an act of desperation reminiscent of the invention of epicycles upon epicycles to shore up the Ptolemaic cosmology).
There are a number of suitable approaches available to the safety analyst, which seek to accommodate epistemic uncertainty (Bayesian methods, Possibility Theory and Dempster-Schafer, to name but three). However, whilst the practitioner is not even aware that there is an issue, and continues to assume the objectivity of all probability, there is little hope that these methods will attract the attention they deserve.
Then, of course, we have to consider the pernicious effect that cognitive bias has upon the analyst’s assessment of likelihood. It is in the nature of such biases that the individual is unaware of their impact. Surely, therefore, even the most basic training in this area would be of considerable benefit to the practitioner. On a similar theme, I have become concerned that the average safety analyst is insufficiently mindful of the distinction to be made between risk aversion and ambiguity aversion. This may lead to a failure to adequately understand the rationality that lies behind a particular decision, but it may also explain why my colleagues didn’t appear to appreciate the importance of undertaking uncertainty management alongside risk management.
Finally, when one considers the interconnectivity of risks, and the complications introduced by multi-stakeholders, it becomes very difficult to think about risk management strategies without having to address ethical issues associated with risk transfer, optimisation and sufficing. But perhaps that is another story.

Luke: Yes, can you say more about the “ethical issues associated with risk transfer, optimization, and sufficing”?

John: In UK health and safety legislation there is an obligation to reduce existing risk levels ‘So Far As Is Reasonably Practicable’ (SFAIRP). This leads to the idea of residual risks being ‘As Low As Reasonably Practicable’ (ALARP). The ALARP concept assumes that an upper limit can be defined, above which the risk is considered ‘intolerable’. In addition, there is a lower limit, below which the risk is considered to be ‘broadly acceptable’. A risk may be allowed to lie between these two limits as long as it can be demonstrated that further reduction would require disproportionate cost and effort. Apart from the vagueness of the terminology used here, the main problem with this view is that it says nothing about the possibility that the management of one risk may bear upon the scale of another. Indeed, one can envisage a network of inter-connected risks in which this knock-on effect will propagate, resulting in the net level of risk increasing (keep in mind that the propagation may include both positive and negative feedback loops). For this reason, there exists the Globally At Least Equivalent (GALE) principle, which holds that upon modifying a system, one must assess the overall, resulting risk level posed by the system rather than focussing purely upon the risk that the modification was intended to address. The idea, of course is that the overall level should never increase.
So far this has all been very basic risk management theory and, on the face of it, the ALARP and GALE principles appear to complement each other. But do they always? Well, in the simple case where all risks are owned and managed by a single authority, this may be the case. But what if the various risks under consideration have differing owners and stakeholders? In such circumstances, parties who own risks and seek to reduce them SFAIRP may find themselves in conflict with each other, with the various stakeholders and with any body that may exist to ensure that the global risk level is not increased.
Perhaps we are now in the province of game theory rather than decision theory. If so, it seems reasonable to insist that the game be played in accordance with ethical constraints, but has anyone declared what these might be? Some seem obvious; for example, never transfer risk to another party without their knowledge and consent. Others may not be so straightforward. I think we are all familiar with the old chestnut of the signalman who can prevent a runaway train from ploughing into a group of schoolchildren by changing the points, but only by causing the certain death of the train driver. Does the signalman have the moral authority to commit murder? Would it be murder whether or not he or she switches the points? If we find this difficult to answer, one can easily envisage similar difficulties when deciding the ethical framework associated with the management of risk collectives.

Luke: Which changes do you most desire for the safety-critical systems industry?

John: I think that there is a lot to be said for making membership of a suitably constructed professional body a legal imperative for undertaking key systems safety engineering roles. Membership would require demonstration of a specified level of competence, adherence to formulated codes of conduct and the adoption of appropriate ideologies. Given my responses to earlier questions, your readers will probably be unsurprised to hear that I hope that this would provide the opportunity to promote a greater understanding of the conceptual framework lying behind the terms ‘risk’ and ‘uncertainty’. In particular, I would like to see a professional promotion of the philosophical, ethical and cognitive dimensions of system safety engineering.
I appreciate that the various engineering disciplines are already well served by a number of professional societies and that, for example, an Independent Safety Assessor (ISA) in the UK would be expected to be a chartered engineer and a recognised expert in the field (whatever that means). However, the epistemic uncertainties surrounding the development and application of complex computer-based systems introduce issues that perhaps the commonly encountered engineering psyche may not be fully equipped to appreciate. It may be that the safety engineering professional may need to think more like a lawyer. Consequently, the professional body I am looking for could be modelled upon the legal profession, as much as upon the existing engineering professions. I know for some people ‘lawyer’ is a dirty word, but so is ‘subjectivity’ in some engineers’ minds. Being pragmatic, and in order to stay in the game, we may all have to become sophists.

Luke: Thanks, John!
The post John Ridgway on safety-critical systems appeared first on Machine Intelligence Research Institute.

John Ridgway on safety-critical systems


David Cook on the VV&A process

Dr. David A. Cook is Associate Professor of Computer Science at Stephen F. Austin State University, where he teaches Software Engineering, Modeling and Simulation, and Enterprise Security. Prior to this, he was Senior Research Scientist and Principal Member of the Technical Staff at AEgis Technologies, working as a Verification, Validation, and Accreditation agent supporting the Airborne Laser. Dr. Cook has over 40 years’ experience in software development and management. He was an associate professor and department research director at USAF Academy and former deputy department head of Software Professional Development Program at AFIT. He has been a consultant for the Software Technology Support Center, Hill AFB, UT for 19 years.
Dr. Cook has a Ph.D. in Computer Science from Texas A&M University, is a Team Chair for ABET, Past President for the Society for Computer Simulation, International, and Chair of ACM SIGAda.

Luke Muehlhauser: In various articles and talks (e.g. Cook 2006), you’ve discussed the software verification, validation, and accreditation (VV&A) process. Though the general process is used widely, the VV&A term is often used when discussing projects governed by DoD 5000.61. Can you explain to whom DoD 5000.61 applies, and how it is used in practice?

David A Cook: DOD 5000.81 applies to all Department of Defense activities involving modeling and simulation.  For all practical purposes, it applies to all models and simulations that are used by the DOD.  This implies that it also applies to all models and simulations created by civilian contractors that are used for DOD purposes.
The purpose of the directive, aside from specifying who is the “accreditation authority” (more on this later) is to require Verification and Validation for all models and simulation, and then also to require that each model and simulation by accredited for its intended use.  This is the critical part, as verification and validation has almost universally been a part of software development within the DOD.  Verification asks the question “Are we building the system in a quality manner?”, or “are we building the system right?”.  Verification, in a model (and the resulting execution of the model providing a simulation) goes a bit further – and asks the question “Does the model build and the results of the simulation actually represent the conceptual design and specifications of the system we built?”  The difference is that in a model and simulation, you have to show that your design and specifications of the system you envision are correctly translated into code, and that the data provided to the code also matches specification.
Validation asks the question “are we building a system that meets the users’ actual needs?”, or “are we building the right system?”  Again, the verification of a model and resulting simulation is a bit more complex than non-M&S ”verification”.  In modeling and simulation, verification has to show that the model and the simulation both accurately represent the “real world” from the perspective of the intended use.
These two activities are extremely difficult when you are building models and providing simulation results for notional systems that might not actually exist in the real world.  For example, it would be difficult to provide V&V for a manned Mars mission, because, in the real world, there is not a manned Mars lander yet!   Therefore, for notional systems, V&V might require estimation and guesswork.  However, guesswork and estimation might be the best you can do!
5000.61 further requires that there be an ultimate authority, the “accreditation authority”, that is willing to say “based on the Verification and Validation performed on this model, I certify that it provides answers that are acceptable for its intended use”.  Again, if you are building a notional system, this requires experts to say “These are guesses, but they are the best guesses available, and the system is as close a model to the real world as possible.  We accredit this system to provide simulation results that are acceptable.”   If, for example, an accredited  simulation shows that a new proposed airplane would be able to carry 100,000 pounds of payload – but the result airplane, once built, can only carry 5,000 pounds – the accreditation authority would certainly bear some of the blame for the problem.
In practice, there are process for providing VV&A.  Military Standard  3022 provides a standard template for recording VV&A activities, and many DOD agencies have their own VV&A repository where common models and simulation VV&A artifacts (and associated documentation) are kept.
There are literally hundreds of ways to verify and validate a model (and it’s associated simulation execution).  The V&V “agents” (who have been tasked with performing V&V) provide a recommendation to the Accreditation Authority, listing what are acceptable uses, and (the critical part) the limits of the model and simulation.  For example, a model and simulation might provide an accurate representation of the propagation of a laser beam (in the upper atmosphere) during daylight hours, but not be a valid simulation at night, due to temperature-related atmospheric propagation.  The same model and simulation might be a valid predictor of a laser bouncing off of a “flat surface”, but not bouncing off of uneven terrain.

Luke: Roughly how many accreditation authorities are there for such projects? Do accreditation authorities tend to specialize in accrediting V&V in certain domains — e.g. some for computer software, some for airplanes, etc.? Are there accreditation authorities that the DoD doesn’t recognize as “legitimate”?

David: Accreditation authorities are simply the component heads who sign a letter saying “Model and Simulation X is approved for the following purposes”.  The letter then states what the intended uses are, lists any special conditions, and lists the limitations of the model and the simulation.  The accreditation authority is more of a position rather than a person.  It can be a person (usually the head of the organization), or a committee.
Each DOD agency is responsible for models and simulations that it develops or uses – they must either VV&A their own models and simulations, or use models and simulations (from a trusted source) that has performed their own VV&A.  Note however, that each DOD agency probably has their own data – which must also be be accredited.  Every project has it’s own M&S, and probably has domain experts who perform the VV&A.  Rather than go to the top for accreditation, each project probably has been delegated the authority to perform its own VV&A.
There are no non-legimate accreditation authorities per se; accreditation authorities are not authorized based on knowledge, simply on position.  However it is assumed that each M&S area has domain experts who have the specialized knowledge in the application area to perform reliable VV&A.  These domain experts span many areas – application domain experts (who might, for example, be an expert on a laser beam), coding domain experts (who can verify that the code is a good representation of the requirements), data domain experts (who verify that the targeting data represents a valid target), and perhaps many others.  Typically, each project has a VV&A team or “agent” who perform the V&V, and recommend accreditation (usually in a formal letter) that restates the intended uses and limitations,  The recommendation includes all associated artifacts, such as test results, reviews, reports of individual Verification and Validation activities, other models and simulation used to compare against, real-life data (to show validity), and possibly many other items.
If a particular DOD agency is using a model and simulation in its exercises, it is responsible for VV&A of its own M&S.  If, on the other hand, an allied agency is using a model that includes artifacts from an another agency – the outside agency is responsible for working to make sure that the model, simulation and data repressing them is valid.  In essence, each DOD agency is responsible to other other DOD component to ensure that their forces and capabilities are appropriately represented to all outside agencies utilizing models and simulation that involve them.

Luke: Are there historical cases in which a model or simulation completed the VV&A process, but then failed to perform as it should have given the accreditation, and then the accreditation authority was held to account for their failure? If so, could you give an example? (Preferably but not necessarily in software.)

David: Because I worked as a consultant on many modeling and simulation projects, I am ethically prevented from discussing actual failures that I know about – mainly because most of the projects I worked on were classified, and I signed non-disclosure agreements.
However, by shifting into hypothetical scenarios, there are several stories that I can use that best illustrate this. One is a story taught in many simulation classes – and I only have secondhand knowledge of it. The other two are ancedotal – but good lessons!
In the first instance, a model was used which predicted “safe runway distance” for an airplane. Feed into the model the weight, altitude, temperature, and humidity, and run the simulation to predict how much runway was needed.
Unfortunately, the day the model was used, it took several hours for the airplane to actually takeoff. It had a bit more fuel that estimated – adding weight. By takeoff time, the temperature had risen, giving “stickier” tires and runway, and decreasing air density (giving less lift) Also – the humidity had changed, also affecting lift characteristics.
The model did not have a large allowance for error (it tried to give a relatively precise answer) – and with all the factors changing, the airplane went from “enough runway” to “marginal” after the simulation had been run. Combined with a relatively inexperienced pilot (who did not advance the throttle fast enough) – and the airplane overshot the end of the runway. Not much damage (other than a bruised ego) – but the simulation – while accurate, was not used properly.
The other two stories are certainly imaginary – but are passed around like legend in our field. In the first story, in the early days of the Airborne Laser, a very simple model was used to predict laser propagation. Code was reused to model the laser – basically, code for a missile, with the speed of the missile increased to the speed of light. The targeting acquisition, target recognition, etc. were all similar, and once fired, the simulation would show if the target was hit. Until the first time they ran it, halfway to the target, the laser beam “ran out of fuel” and fell into the sea.
The second (certainly imaginary) story involves modeling a battle scenario for the Australian Air Force – using helicopters. One of the problems with landing a helicopter was making sure it had a clear landing field – and kangaroos were a problem. So – the developers of the battle simulation, who used object-oriented development, took some code which was basically used to model a ground soldier and modified its behavior to “run at the sound of helicopters”. They then changed its appearance on the simulation to show a small image of a kangaroo. When to model was executed, the simulation showed the kangaroos running away from the helicopter. Until it landed, and then the kangaroos reversed direction, and attacked the helicopter with rifles!
Ok – the last two examples are cute and funny, but show the problems with invalid assumptions and imperfect data.
I leave this question with a quote that I always use in my M&S classes – “All models are inaccurate. Some models are useful anyway.” It is extremely difficult to model the real world totally inside of code. I don’t care how well you model a hot tub in a simulation – it is NOT really a hot tub.
There are always things you do not consider, data that is not perfect, or constraints that you miss. A model is a best-guess approximation of what will happen in the “ dal world” – but it is NOT the “real world”. All models have limitations. In spite of that. the resulting model and the simulation still give useful data. The accreditation authority is simply acknowledging that the model and simulation are useful “for their intended use”, and that “limitations exist”. No reputable modeling and simulation expert (nor any accreditation authority) trusts a single model and its resulting simulation to produce data that is used in life-or-death decisions. Multiple sources of validity are required, multiple independently-developed models and simulations are used, and domain experts are consulted to see if the results “feel right”. And tolerances must always be given. An aircraft might encounter a puddle of water when trying to takeoff. It might hit a bird. Both of these decrease speed, requiring longer takeoff distance. It’s hard to model unforeseen circumstances. If you include a “fudge factor” – how much “fudge factor” is correct? Before an accreditation authority accepts a model or simulation as reliable, many, many steps must be taken to make sure that it produces credible results, and equally as many steps must be taken to make sure that the limitations of the model and simulation are listed and observed before accepting the result of the simulation as valid.

Luke: How did the VV&A process develop at DoD? When did it develop? Presumably it developed in one or more domains first, and then spread to become a more universal expectation?

David: Interesting question. Before we can discuss VV&A, we have to take a slight detour through the history of M&S. And I need to tie several threads of though together.
VV&A is, of course, tied to the use of models and simulations. To be honest, the VV&A of models goes back to the civil war (and probably earlier) – when mathematical models were used to predict firing data (given desired range, here was amount of powder and elevation required. Obviously – the models needed a lot of V&V. However, all it too to V&V the model was to load a cannon and fire it. Not a complex process. The accreditation part was implicit – the Secretary of War used to “authorize” the data to be printed. To really need VV&A, however, complex simulations were needed – and it took computing power to achieve complex M&S.
Over the years, modeling became more and more important, as models and simulation were used for problems that could not easily be solved by traditional mathematical methods. To quote from Wikipedia article,
Computer simulation developed hand-in-hand with the rapid growth of the computer, following its first large-scale deployment during the Manhattan Project in World War II to model the process of nuclear detonation. It was a simulation of 12 hard spheres using a Monte Carlo algorithm. Computer simulation is often used as an adjunct to, or substitute for, modeling systems for which simple closed form analytic solutions are not possible. There are many types of computer simulations; their common feature is the attempt to generate a sample of representative scenarios for a model in which a complete enumeration of all possible states of the model would be prohibitive or impossible.
VV&A really did not become a serious issue until models and simulations were computerized, and computers did not become available until the late 1940s. Starting in the late 1940s, both digital and analog computers became available. However, very few (if any) engineers were trained on how to use this newly-developed computing power. There are various stories of how modeling and simulation became a powerful force in the DOD, but the story I have personal knowledge of is the story of John McLeod – an engineer working at the Naval Air Missile Test Center at Point Mugu on the California coast north of Los Angeles. John was an innovator and after working on analog computers and simulations in the early 1950s, , was John McLeod, who took delivery of a new analog computer sometime in 1952. John was not the only engineer in the aerospace community in Southern California facing the same problems, and a few of them decided to get together as an informal user group to exchange ideas and experiences. To make a long story short, John helped found what became the Society for Computer Simulation (SCS). This organization, over the years, has had members who were leaders and innovators in the field of modeling, simulation, and VV&A. [Note that I had the privilege to be the President of SCS from 2011 – 2012, so I am a bit biased]. The SCS has, to this day, the McLeod award to commemorate the advances John McLeod made in the M&S arena. It is only awarded to those that have made significant contributions to the profession.
The SCS published newsletters. M&S conferences were organized. Leaders in the field were able to meet, publish, and share their expertise. All of which help integrate M&S into more and more domains. As a result of leaders in the field being able to share M&S information, and also as a result huge increase in capabilities and availability of computers to run M&S, the need for VV&A also increased. Over the years, modeling and simulation became more and more important in many domains within the DOD. It helped develop fighters (in fact, aircraft of all types). It helped train our astronauts to land on the moon. It modeled the space shuttle. Complex models and simulations helped us model ballistic missile defense, fight war games with minimal expense (and no lives lost!), and design complex weapon systems. In fact, it’s hard to imagine any technologically sophisticated domain that does not use M&S to save money, save time, and ensure safety. But – these increasingly complex models needed verification and validation, and frequently accreditation,
So – the proliferation in the use of M&S lead to an increased need for VV&A. M&S became so complex that VV&A could not be accomplished without “domain experts” – usually referred to as “Subject Matter Experts” (SMEs) to help. Increased complexity of the M&S lead to increased complexity of the VV&A. Various elements within the DOD were performing VV&A on their own, with little official coordination. To leverage the experience of various DOD components and multiple domains, the DOD saw the need for a single point of coordination. As a result, in the 1990s, the DOD formed the Defense Modeling and Simulation Office (DMSO). The DMSO served as a single point of coordination for all M&S (and VV&A) efforts within the DOD. One of the best DMSO contributions was the VV&A Recommended Practices Guide (VV&A RPG) – first published in 1996. The guide has been updated several times over the years, reflecting the uncreased importance of VV& in the DOD. In 2008 DMSO was renamed the Modeling and Simulation Coordination Office. The MSCO web site (and the latest version of the VV&A Recommended Practices Guide) can be found at msco.mil
For those of you interested in M&S and VV&A, I cannot recommend the MSCO resource enough. It costs nothing (not even email registration), and contains a vast amount of information about M&S and VV&A. The RPG Key Concepts document alone contains 34 pages of critical “background” information that you should read before going any further in VV&A.

Luke: In Cook (2006) you write that one of the reasons V&V is so difficult comes from “having to ‘backtrack’ and fill in blanks long after development.” What does this mean? Can you give an example?

David: Let’s imagine you are designing a new fighter aircraft. It is still on the drawing board, and only plans exist.
Rather than spend money building an actual prototype first, you develop mathematical models of the jet to help verify the performance characteristics. You might actually build a very small model of the body – maybe 1/10th size for wind tunnel experiments.
You also build computer-based models and execute them to estimate flight characteristics. The wind-tunnel experiences (even though only on a 1/10th size model) will give data that might make you modify or change the computer-based model. This feedback loop consists of “build model – run simulation – examine data – adjust model” and repeat.
Eventually, you build a working prototype of the jet. Almost certainly, the actual flight characteristics will not exactly match the computer-based model. The prototype is “real world” – so you have to readjust the computer-based model. The “real-world” prototype is just a prototype – and probably not used for high-speed fighting and turns – but the basic data gathered from the flying of the prototype leads to changes in the computer-based model, which will now be used to predict more about high-speed maneuvering.
Back when I worked on the Airborne Laser – we had models that predicted the laser performance before the laser was actually built or fired! The models were based on mathematical principles, on data from other lasers, and from simpler, earlier models that were being improved on. Once a working Airborne Laser was built and fired – we had “real world” data. It was no surprise to find out that the actual characteristics of the laser beam were slightly different that those predicted by the models. For one thing, the models were simplistic – it was impossible to take everything into account. The result was that we took the real-world data, and modified the computer models to permit them to better predict future performance.
The bottom line is that the model is never finished. Every time you have additional data from the “real world” that is not an exact match to what the model predicts, the model should be examined, and the model adjusted as necessary.
There are two terms I like to use for models when it comes to VV&A – “anchoring” and “benchmarking”. If I can get another independently-developed model to predict the same events as my model, I have a source of validation. I refer this as benchmarking. Subject matter experts, other simulations, similar events that lend credence to your model – all improve the validity, and provide benchmarking. Anchoring, on the other hand, is when I tie my model directly to real-world data.
As long as the model is being used to predict behavior – it needs to continually be tied or anchored to real-world performance, if possible. If no real-world data is available, then similar models, expert opinions, etc. can be used to also increase the validity.
Just a final note. Models can become so engrained in thoughts that they become “real. For example, I remember when the recent Star Trek movie (the 2009 version) came out. A friend of mine said, after viewing the movie, that he had trouble the the bridge of the USS Enterprise. It did not “look real”. I asked what “real” was – and my friend replied “You know, like the REAL USS Enterprise, the NCC 1701 (referring to the original series). Think about it – all are notional and imaginary (sorry, fellow Trekers) – yet he viewed one as “real” and the other as inaccurate. Models – when no real-world artifact exists – have the potential to become “real” in your mind. It’s worth remembering that a model is NOT real, but only an artifact built to resemble or predict what might (or might not) eventually become real one day.

Luke: Do you have a sense of how common formal verification is for software used in DoD applications? Is formal verification of one kind or another required for certain kinds of software projects? (Presumably, DoD also uses much software that is not amenable to formal methods.)

David: I have not worked on any project that uses formal V&V methods.
I used to teach the basics of formal methods (using ‘Z’- pronounced Zed) – but it is very time consuming, and not really fit for a lot of project.
Formal notation shows the correctness of the algorithm from a mathematical standpoint. For modeling and simulation, however, they do not necessarily help you with accreditation – because the formal methods check the correctness of the code, and not necessarily the correlation of the cod eight real-world data.
I have heard that certain extremely critical applications (such as reactor code, and code for the Martian Lander) use formal methods to make sure that the code is correct. However, formal methods take a lot of training and education to use correctly, and it also consumes a lot of time in actual use. Formal methods seldom (never?) speed up the process – they are strictly used to validate the code.
From my experience, I have not work on on any project that made any significant use of formal methods – and in fact, I do not have any colleagues that have used formal methods, either.

Luke: Thanks, David!
The post David Cook on the VV&A process appeared first on Machine Intelligence Research Institute.

David Cook on the VV&A process