MIRI's September Newsletter

Thanks to the generosity of 100+ donors, we successfully completed our 2014 summer matching challenge on August 15th, raising more than $400,000 total for our research program. Our deepest thanks to all our supporters!

Research updates

New paper: "Exploratory engineering in artificial intelligence."
2 new analyses: "Groundwork for AGI safety engineering" and "Loosemore on AI safety and attractors."

News updates

MIRI is running an online reading group for Nick Bostrom's Superintelligence. Join the discussion here!
MIRI participated in the 2014 Effective Altruism Summit. Slides from our talks are available here.

Other updates

Nick Bostrom is touring the US and Europe to discuss Superintelligence. Dates are available on his website. He visits Berkeley on September 12th; details here.
Paul Christiano and Katja Grace have launched a new website containing many analyses related to the long-term future of AI: AI Impacts.
Paul Christiano has published several additional analyses closely related to MIRI's Friendly AI interests: "Three impacts of machine intelligence," "Approval-seeking [agents]," "Straightforward vs. goal-oriented communication," "Specifying a human precisely," "Specifying enlightened judgment precisely," "Challenges for extrapolation," and "Confronting Gödelian difficulties."
The Hanson-Yudkowsky AI Foom Debate continues with Hanson's post "I Still Don't Get Foom."

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's September Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s September Newsletter


Superintelligence reading group

Nick Bostrom's eagerly awaited Superintelligence comes out in the US this week. To help you get the most out of it, MIRI is running an online reading group where you can join with others to ask questions, discuss ideas, and probe the arguments more deeply.
The reading group will "meet" on a weekly post on the LessWrong discussion forum. For each 'meeting', we will read about half a chapter of Superintelligence, then come together virtually to discuss. I'll summarize the chapter, and offer a few relevant notes, thoughts, and ideas for further investigation. (My notes will also be used as the source material for the final reading guide for the book.)
Discussion will take place in the comments. I'll offer some questions, and invite you to bring your own, as well as thoughts, criticisms and suggestions for interesting related material. Your contributions to the reading group might also (with permission) be used in our final reading guide for the book.
We welcome both newcomers and veterans on the topic. Content will aim to be intelligible to a wide audience, and topics will range from novice to expert level. All levels of time commitment are welcome. We especially encourage AI researchers and practitioners to participate. Just use a pseudonym if you don't want your questions and comments publicly linked to your identity.
We will follow this preliminary reading guide, produced by MIRI, reading one section per week.
If you have already read the book, don't worry! To the extent you remember what it says, your superior expertise will only be a bonus. To the extent you don't remember what it says, now is a good time for a review! If you don't have time to read the book, but still want to participate, you are also welcome to join in. I will provide summaries, and many things will have page numbers, in case you want to skip to the relevant parts.
If this sounds good to you, first grab a copy of Superintelligence. You may also want to sign up here to be emailed when the discussion begins each week. The first virtual meeting (forum post) will go live at 6pm Pacific on Monday, September 15th. Following meetings will start at 6pm every Monday, so if you'd like to coordinate for quick fire discussion with others, put that into your calendar. If you prefer flexibility, come by any time! And remember that if there are any people you would especially enjoy discussing Superintelligence with, link them to this post!
Topics for the first week will include impressive displays of artificial intelligence, why computers play board games so well, and what a reasonable person should infer from the agricultural and industrial revolutions.

The post Superintelligence reading group appeared first on Machine Intelligence Research Institute.

Superintelligence reading group


New paper: "Exploratory engineering in artificial intelligence"

Luke Muehlhauser and Bill Hibbard have a new paper (PDF) in the September 2014 issue of Communications of the ACM, the world's most-read peer-reviewed computer science publication. The title is "Exploratory Engineering in Artificial Intelligence."
Excerpt:
We regularly see examples of new artificial intelligence (AI) capabilities… No doubt such automation will produce tremendous economic value, but will we be able to trust these advanced autonomous systems with so much capability?
Today, AI safety engineering mostly consists in a combination of formal methods and testing. Though powerful, these methods lack foresight: they can be applied only to particular extant systems. We describe a third, complementary approach that aims to predict the (potentially hazardous) properties and behaviors of broad classes of future AI agents, based on their mathematical structure (for example, reinforcement learning)… We call this approach "exploratory engineering in AI."
…
In this Viewpoint, we focus on theoretical AI models inspired by Marcus Hutter's AIXI, an optimal agent model for maximizing an environmental reward signal…
…
Autonomous intelligent machines have the potential for large impacts on our civilization. Exploratory engineering gives us the capacity to have some foresight into what these impacts might be, by analyzing the properties of agent designs based on their mathematical form. Exploratory engineering also enables us to identify lines of research — such as the study of Dewey's value-learning agents — that may be important for anticipating and avoiding unwanted AI behaviors. This kind of foresight will be increasingly valuable as machine intelligence comes to play an ever-larger role in our world.
The post New paper: "Exploratory engineering in artificial intelligence" appeared first on Machine Intelligence Research Institute.

New paper: “Exploratory engineering in artificial intelligence”


2014 Summer Matching Challenge Completed!

Thanks to the generosity of 100+ donors, today we successfully completed our 2014 summer matching challenge, raising more than $400,000 total for our research program.
Our deepest thanks to all our supporters!
Also, Jed McCaleb's new crypto-currency Stellar was launched during MIRI's fundraiser, and we decided to accept donated stellars. These donations weren't counted toward the matching drive, and their market value is unstable at this early stage, but as of today we've received 850,000+ donated stellars from 3000+ different stellar accounts. Our thanks to everyone who donated in stellar!
The post 2014 Summer Matching Challenge Completed! appeared first on Machine Intelligence Research Institute.


MIRI's recent effective altruism talks

MIRI recently participated in the 2014 Effective Altruism Retreat and Effective Altruism Summit organized by Leverage Research. We gave four talks, participated in a panel, and held "office hours" during which people could stop by and ask us questions.
The slides for our talks are available below:

Muehlhauser, MIRI Intro
Yudkowsky, Caring about Many Distant People in Strange Situations [no slides]
Muehlhauser, Steering the Future of AI
Yudkowsky, Why We're Doing It and What MIRI is Doing

If videos of these talks become available, we'll link them from here as well.
See also our earlier posts Friendly AI Research as Effective Altruism and Why MIRI?
The post MIRI's recent effective altruism talks appeared first on Machine Intelligence Research Institute.

MIRI’s recent effective altruism talks


Groundwork for AGI safety engineering

Improvements in AI are resulting in the automation of increasingly complex and creative human behaviors. Given enough time, we should expect artificial reasoners to begin to rival humans in arbitrary domains, culminating in artificial general intelligence (AGI).
A machine would qualify as an 'AGI', in the intended sense, if it could adapt to a very wide range of situations to consistently achieve some goal or goals. Such a machine would behave intelligently when supplied with arbitrary physical and computational environments, in the same sense that Deep Blue behaves intelligently when supplied with arbitrary chess board configurations — consistently hitting its victory condition within that narrower domain.
Since generally intelligent software could help automate the process of thinking up and testing hypotheses in the sciences, AGI would be uniquely valuable for speeding technological growth. However, this wide-ranging productivity also makes AGI a unique challenge from a safety perspective. Knowing very little about the architecture of future AGIs, we can nonetheless make a few safety-relevant generalizations:

Because AGIs are intelligent, they will tend to be complex, adaptive, and capable of autonomous action, and they will have a large impact where employed.
Because AGIs are general, their users will have incentives to employ them in an increasingly wide range of environments. This makes it hard to construct valid sandbox tests and requirements specifications.
Because AGIs are artificial, they will deviate from human agents, causing them to violate many of our natural intuitions and expectations about intelligent behavior.

Today's AI software is already tough to verify and validate, thanks to its complexity and its uncertain behavior in the face of state space explosions. Menzies & Pecheur (2005) give a good overview of AI verification and validation (V&V) methods, noting that AI, and especially adaptive AI, will often yield undesired and unexpected behaviors.
An adaptive AI that acts autonomously, like a Mars rover that can't be directly piloted from Earth, represents an additional large increase in difficulty. Autonomous safety-critical AI agents need to make irreversible decisions in dynamic environments with very low failure rates. The state of the art in safety research for autonomous systems is improving, but continues to lag behind system capabilities work. Hinchman et al. (2012) write:
As autonomous systems become more complex, the notion that systems can be fully tested and all problems will be found is becoming an impossible task. This is especially true in unmanned/autonomous systems. Full test is becoming increasingly challenging on complex system. As these systems react to more environmental [stimuli] and have larger decision spaces, testing all possible states and all ranges of the inputs to the system is becoming impossible. […] As systems become more complex, safety is really risk hazard analysis, i.e. given x amount of testing, the system appears to be safe. A fundamental change is needed. This change was highlighted in the 2010 Air Force Technology Horizon report, "It is possible to develop systems having high levels of autonomy, but it is the lack of suitable V&V methods that prevents all but relatively low levels of autonomy from being certified for use." […]
The move towards more autonomous systems has lifted this need [for advanced verification and validation techniques and methodologies] to a national level.
AI acting autonomously in arbitrary domains, then, looks particularly difficult to verify. If AI methods continue to see rapid gains in efficiency and versatility, and especially if these gains further increase the opacity of AI algorithms to human inspection, AI safety engineering will become much more difficult in the future. In the absence of any reason to expect a development in the lead-up to AGI that would make high-assurance AGI easy (or AGI itself unlikely), we should be worried about the safety challenges of AGI, and that worry should inform our research priorities today.
Below, I'll give reasons to doubt that AGI safety challenges are just an extension of narrow-AI safety challenges, and I'll list some research avenues people at MIRI expect to be fruitful.

New safety challenges from AGI
A natural response to the idea of starting work on high-assurance AGI is that AGI itself appears to be decades away. Why worry about it now? And, supposing that we should worry about it, why think there's any useful work we can do on AGI safety so far in advance?
In response to the second question: It's true that, at a glance, AGI looks difficult to effectively prepare for. The issue is important enough, however, to warrant more than a glance. Long-term projects like mitigating climate change or detecting and deflecting asteroids are intuitively difficult. The same is true of interventions that depend on projected future technologies, such as work on post-quantum cryptography in anticipation of quantum computers. In spite of that, we've made important progress on these fronts.
Covert channel communication provides one precedent. It was successfully studied decades in advance of being seen in the wild. Roger Schell cites a few other success cases in Muehlhauser (2014b), and suggests reasons why long-term security and safety work remains uncommon. We don't know whether early-stage AGI safety work will be similarly productive, but we shouldn't rule out the possibility before doing basic research into the question. I'll list some possible places to start looking in the next section.
What about the first question? Why worry specifically about AGI?
I noted above that AGI is an extreme manifestation of many ordinary AI safety challenges. However, MIRI is particularly concerned with unprecedented, AGI-specific behaviors. For example: An AGI's problem-solving ability (and therefore its scientific and economic value) depends on its ability to model its environment. This includes modeling the dispositions of its human programmers. Since the program's success depends in large part on its programmers' beliefs and preferences, an AI pursuing some optimization target can select actions for the effect they have on programmers' mental states, not just on the AI's material environment.
This means that safety protocols will need to be sensitive to risks that differ qualitatively from ordinary software failure modes — AGI-specific hazards like 'the program models its programmed goals as being better served if it passes human safety inspections, so it selects action policies that make it look safer (to humans) than it really is'. If we model AGI behavior using only categories from conventional software design, we risk overlooking new intelligent behaviors, including 'deception'.
At the same time, oversimplifying these novel properties can cause us to anthropomorphize the AGI. If it's naïve to expect ordinary software validation methods to immediately generalize to advanced autonomous agents, it's even more naïve to expect conflict prevention strategies that work on humans to immediately generalize to an AI. A 'deceptive' AGI is just one whose planning algorithm identifies some human misconception as instrumentally useful to its programmed goals. Its methods or reasons for deceiving needn't resemble a human's, even if its capacities do.
In human society, we think, express, and teach norms like 'don't deceive' or Weld & Etzioni's (1994) 'don't let humans come to harm' relatively quickly and easily. The complex conditional response that makes humans converge on similar goals remains hidden inside a black box — the undocumented spaghetti code that is the human brain. As a result of our lack of introspective access to how our social dispositions are cognitively and neurally implemented, we're likely to underestimate how contingent and complex they are. For example:

We might expect that an especially intelligent AI system would have especially worthy goals, since knowledge and insight are associated with many other virtues in humans. E.g., Halls (2007) conjectures this on the grounds that criminality negatively correlates with intelligence in humans. Bostrom's (2003) response is that there's no particular reason to expect AIs to converge on anthropocentric terminal values like 'compassion' or 'loyalty' or 'novelty'. A superintelligent AI could consistently have no goal other than to construct paperclips, for example.
We might try to directly hand-code goals like 'don't deceive' into the agent by breaking the goals apart into simpler goals, e.g., 'don't communicate information you believe to be false'. However, in the process we're likely to neglect the kinds of subtleties that can be safely left implicit when it's a human child we're teaching — lies of omission, misleading literalism, novel communication methods, or any number of edge cases. As Bostrom (2003) notes, the agent's goals may continue to reflect the programmers' poor translation of their requirements into lines of code, even after its intelligence has arrived at a superior understanding of human psychology.
We might instead try to instill the AGI with humane values via machine learning — training it to promote outcomes associated with camera inputs of smiling humans, for example. But a powerful search process is likely to hit on solutions that would never occur to a developing human. If the agent becomes more powerful or general over time, initially benign outputs may be a poor indicator of long-term safety.

Advanced AI is also likely to have technological capabilities, such as strong self-modification, that introduce other novel safety obstacles; see Yudkowsky (2013).
These are quick examples of some large and poorly-understood classes of failure mode. However, the biggest risks may be from problem categories so contrary to our intuitions that they will not occur to programmers at all. Relying on our untested intuitions, or on past experience with very different systems, is unlikely to catch every hazard.
As an intelligent but inhuman agent, AGI represents a fundamentally new kind of safety challenge. As such, we'll need to do basic theoretical work on the general features of AGI before we can understand such agents well enough to predict and plan for them.

 Early steps
What would early, theoretical AGI safety research look like? How does one vet a hypothetical technology? We can distinguish research projects oriented toward system verification from projects oriented toward system requirements.
Verification-directed AGI research extends existing AI safety and security tools that are likely to help confirm various features of advanced autonomous agents. Requirements-directed AGI research instead specifies desirable AGI abilities or behaviors, and tries to build toy models exhibiting the desirable properties. These models are then used to identify problems to be overcome and basic gaps in our conceptual understanding.
In other words, verification-directed approaches would ask 'What tools and procedures can we use to increase our overall confidence that the complex systems of the future will match their specifications?' They include:

Develop new tools for improving the transparency of AI systems to inspection. Frequently, useful AI techniques like boosting are tried out in an ad-hoc fashion and then promoted when they're observed to work on some problem set. Understanding when and why programs work would enable stronger safety and security guarantees. Computational learning theory may be useful here, for proving bounds on the performance of various machine learning algorithms.
Extend techniques for designing complex systems to be readily verified. Work on clean-slate hardware and software approaches that maintain high assurance at every stage, like HACMS and SAFE.
Extend current techniques in program synthesis and formal verification, with a focus on methods applicable to complex and adaptive systems, such as higher-order program verification and Spears' (2000, 2006) incremental reverification. Expand on existing tools — e.g., design better interfaces and training methods for the SPIN model checker to improve its accessibility.
Apply homotopy type theory to program verification. The theory's univalence axiom lets us derive identities from isomorphisms. Harper & Licata (2011) suggest that if we can implement this as an algorithm, it may allow us to reuse high-assurance code in new contexts without a large loss in confidence.
Expand the current body of verified software libraries and compilers, such as the Verified Software Toolchain. A lot of program verification work is currently directed at older toolchains, e.g., relatively small libraries in C. Focusing on newer toolchains would limit our ability to verify systems that are already in wide use, but would put us in a better position to verify more advanced safety-critical systems.

Requirements-directed approaches would ask 'What outcomes are we likely to want from an AGI, and what general classes of agent could most easily get us those outcomes?' Examples of requirements-directed work include:

Formalize stability guarantees for intelligent self-modifying agents. A general intelligence could help maintain itself and implement improvements to its own software and hardware, including improvements to its search and decision heuristics à la EURISKO. It may be acceptable for the AI to introduce occasional errors in its own object recognition or speech synthesis modules, but we'd want pretty strong assurance about the integrity of its core decision-making algorithms, including the module that approves self-modifications. At present, toy models of self-modifying AI discussed in Fallenstein & Soares (2014) run into two paradoxes of self-reference, the 'Löbian obstacle' and the 'procrastination paradox.' Similar obstacles may arise for real-world AGI, and finding solutions should improve our general understanding of systems that make decisions and predictions about themselves.
Specify desirable checks on AGI behavior. Some basic architecture choices may simplify the task of making AGI safer, by restricting the system's output channel (e.g., oracle AI in Armstrong et al. (2012)) or installing emergency tripwires and fail-safes (e.g., simplex architectures). AGI checks are a special challenge because of the need to recruit the agent to help actively regulate itself. If this demand isn't met, the agent may devote its problem-solving capabilities to finding loopholes in its restrictions, as in Yampolskiy's (2012) discussion of an 'AI in a box'.
Design general-purpose methods by which intelligent agents can improve their models of user requirements over time. The domain of action of an autonomous general intelligence is potentially unlimited; or, if it is limited, we may not be able to predict which limits will apply. As such, it needs safe situation-general goals. Coding a universally safe and useful set of decision criteria by hand, however, looks hopelessly difficult. Instead, some indirectly normative method like Dewey's (2011) value learning seems necessary, to allow initially imperfect decision-makers to improve their goal content over time. Related open questions include: what kinds of base cases can we use to train and test a beneficial AI?; and how can AIs be made safe and stable during the training process?
Formalize other optimality criteria for arbitrary reasoners. Just as a general-purpose adaptive agent would need general-purpose values, it would also need general-purpose methods for tracking features of its environment and of itself, and for selecting action policies on this basis. Mathematically modeling ideal inference (e.g., Hutter (2012)), ideal decision-theoretic expected value calculation (e.g., Altair (2013)), and ideal game-theoretic coordination (e.g., Barasz et al. (2014)) are unlikely to be strictly necessary for AGI. All the same, they're plausibly necessary for AGI safety, because models like these would give us a solid top-down theoretical foundation upon which to cleanly construct components of autonomous agents for human inspection and verification.

We can probably make progress in both kinds of AGI safety research well in advance of building a working AGI. Requirements-directed research focuses on abstract mathematical agent models, which makes it likely to be applicable to a wide variety of software implementations. Verification-directed approaches will be similarly valuable, to the extent they are flexible enough to apply to future programs that are much more complex and dynamic than any contemporary software. We can compare this to present-day high-assurance design strategies, e.g., in Smith & Woodside (2000) and Hinchman et al. (2012). The latter write, concerning simpler autonomous machines:
With better software analysis techniques, software can be analyzed at design-time with the goal of finding software faults earlier. This analysis can also prove the absence of error or negative properties. As system complexity and functionality increase, complete testing is becoming impossible and enhanced analysis techniques will have to be used. Furthermore, many of these software techniques, such as model checking, can be used in analysis of requirements and system design to find conflicting requirements or logic faults before a single line of code is written saving more time and money over traditional testing methods.
Verification- and requirements-directed work is complementary. The point of building clear mathematical models of agents with desirable properties is to make it easier to design systems whose behaviors are transparent enough to their programmers to be rigorously verified; and verification methods will fail to establish system safety if we have a poor understanding of what kind of system we want in the first place.
Some valuable projects will also fall in between these categories — e.g., developing methods for principled formal validation, which can increase our confidence that we're verifying the right properties given the programmers' and users' goals. (See Cimatti et al. (2012) on formal validation, and also Rushby (2013) on epistemic doubt.)

 MIRI's focus: the mathematics of intelligent agents
MIRI's founder, Eliezer Yudkowsky, has been one of the most vocal advocates for research into autonomous high-assurance AGI, or 'friendly AI'. Russell & Norvig (2009) write:
[T]he challenge is one of mechanism design — to define a mechanism for evolving Al systems under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes. We can't just give a program a static utility function, because circumstances, and our desired responses to circumstances, change over time.
MIRI's approach is mostly requirements-directed, in part because this angle of attack is likely to enhance our theoretical understanding of the entire problem space, improving our research priorities and other strategic considerations. Moreover, the relevant areas of theoretical computer science and mathematics look much less crowded. There isn't an established subfield or paradigm for requirements-directed AGI safety work where researchers could find a clear set of open problems, publication venues, supervisors, or peers.
Rather than building on current formal verification methods, MIRI prioritizes jump-starting these new avenues of research. Muehlhauser (2013) writes that engineering innovations often have their germ in prior work in mathematics, which in turn can be inspired by informal philosophical questions. At this point, AGI safety work is just beginning to enter the 'mathematics' stage. Friendly AI researchers construct simplified models of likely AGI properties or subsystems, formally derive features of those models, and check those features against general or case-by-case norms.
Because AGI safety is so under-researched, we're likely to find low-hanging fruit even in investigating basic questions like 'What kind of prior probability distribution works best for formal agents in unknown environments?' As Gerwin Klein notes in Muehlhauser (2014a), "In the end, everything that makes it easier for humans to think about a system, will help to verify it." And, though MIRI's research agenda is determined by social impact considerations, it is also of general intellectual interest, bearing on core open problems in theoretical computer science and mathematical logic.
At the same time, it's important to keep in mind that formal proofs of AGI properties only function as especially strong probabilistic evidence. Formal methods in computer science can decrease risk and uncertainty, but they can't eliminate it. Our assumptions are uncertain, so our conclusions will be too.
Though we can never reach complete confidence in the safety of an AGI, we can still decrease the probability of catastrophic failure. In the process, we are likely to come to a better understanding of the most important ways AGI can defy our expectations. If we begin working now to better understand AGI as a theoretical system, we'll be in a better position to implement robust safety measures as AIs improve in intelligence and autonomy in the decades to come.

Acknowledgments
My thanks to Luke Muehlhauser, Shivaram Lingamneni, Matt Elder, Kevin Carlson, and others for their feedback on this piece.

References

Altair (2013). A comparison of decision algorithms on Newcomblike problems. Machine Intelligence Research Institute.
Armstrong et al. (2012). Thinking inside the box: controlling and using an oracle AI. Minds and Machines, 22: 299-324.
Barasz et al. (2014). Robust cooperation in the Prisoner's Dilemma: program equilibrium via probability logic. arXiv.
Bostrom (2003). Ethical issues in advanced artificial intelligence. In Smith et al. (eds.), Cognitive, Emotive and Ethical Aspects of Decision-Making in Humans and in Artificial Intelligence, 2: 12-17.
Cimatti et al. (2012). Validation of requirements for hybrid systems: a formal approach. ACM Transactions on Software Engineering and Methodology, 21.
Dewey (2011). Learning what to value. Artificial General Intelligence 4th International Conference Proceedings: 309-314.
Fallenstein & Soares (2014). Problems of self-reference in self-improving space-time embedded intelligence. Working paper.
Halls (2007). Beyond AI: Creating the Conscience of the Machine.
Harper & Licata (2011). Foundations and applications of higher-dimensional directed type theory. National Science Foundation grant proposal.
Hinchman et al. (2012). Towards safety assurance of trusted autonomy in Air Force flight-critical systems. Layered Assurance Workshop, 17.
Hutter (2012). One decade of Universal Artificial Intelligence. Theoretical Foundations of Artificial General Intelligence, 4: 67-88.
Menzies & Pecheur (2005). Verification and validation and artificial intelligence. Advances in Computers, 65: 154-203.
Muehlhauser (2013). From philosophy to math to engineering. MIRI Blog.
Muehlhauser (2014a). Gerwin Klein on formal methods. MIRI Blog.
Muehlhauser (2014b). Roger Schell on long-term computer security research. MIRI Blog.
Rushby (2013). Logic and epistemology in safety cases. Computer Safety, Reliability, and Security: Proceedings of SafeComp 32 (pp. 1-7).
Russell & Norvig (2009). Artificial Intelligence: A Modern Approach.
Smith & Woodside (2000). Performance validation at early stages of software development. In Gelenbe (ed.), System Performance Evaluation: Methodologies and Applications (pp. 383-396).
Spears (2000). Asimovian adaptive agents. Journal of Artificial Intelligence Research, 13: 95-153.
Spears (2006). Assuring the behavior of adaptive agents. In Rouff et al. (eds.), Agent Technology from a Formal Perspective (pp. 227-257).
Weld & Etzioni (1994). The First Law of Robotics (a call to arms). Proceedings of the Twelfth National Conference on Artificial Intelligence: <PHONE>.
Yampolskiy (2012). Leakproofing the singularity: artificial intelligence confinement problem. Journal of Consciousness Studies, 19: 194-214.
Yudkowsky (2013). Intelligence explosion microeconomics. Technical report.

The post Groundwork for AGI safety engineering appeared first on Machine Intelligence Research Institute.

Groundwork for AGI safety engineering


MIRI's August 2014 newsletter

Dear friends,

Our summer matching challenge is underway! Every donation made to MIRI between now and August 15th, 2014 will be matched dollar-for-dollar, up to a total of $200,000!
Please donate now to help support our research!
Research Updates

May 2015: Cambridge decision theory workshop
2 new analyses: "Some studies which could improve our strategic picture of superintelligence" and "Tentative tips for… prediction or forecasting"
1 new expert interview: Scott Frickel on intellectual movements.

News Updates

There are now 10 MIRIx groups around the world, in 4 countries.
Nick Bostrom will speak about his new book Superintelligence at UC Berkeley on September 12th.
Jed McCaleb has launched a new digital currency, stellars. MIRI now accepts donated stellars; our public name for receiving stellars is: miri

Other Updates

Nick Bostrom's new book Superintelligence has been released in the UK, and the Kindle version is available in the US. (Hardcopy available in the US on Sep. 1st.)

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's August 2014 newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s August 2014 newsletter


Scott Frickel on intellectual movements

 Scott Frickel is Associate Professor in the Department of Sociology and Institute for the Study of Environment and Society at Brown University. His research interweaves sociological analysis with environmental studies and science and technology studies. Prior to coming to Brown he was Boeing Distinguished Professor of Environmental Sociology at Washington State University. He holds a Ph.D. from the University of Wisconsin – Madison.
His research has appeared in a wide range of disciplinary and interdisciplinary journals, including American Sociological Review; Annual Review of Sociology; Science, Technology and Human Values; and Environmental Science and Policy. He is author of Chemical Consequences: Environmental Mutagens, Scientist Activism, and the Rise of Genetic Toxicology and co-editor with Kelly Moore of The New Political Sociology of Science: Institutions, Networks, and Power.

Luke Muehlhauser: In Frickel & Gross (2005), you and your co-author present a "general theory" of scientific/intellectual movements (SIMs). I'll summarize the theory briefly for our readers. In your terminology:

"SIMs have a more or less coherent program for scientific or intellectual change… toward whose knowledge core participants are consciously oriented…"
"The aforementioned core consists of intellectual practices that are contentious relative to normative expectations within a given… intellectual domain."
"Precisely because the intellectual practices recommended by SIMs are contentious, SIMs are inherently political… because every program for intellectual change involves a desire to alter the configuration of social positions within or across intellectual fields in which power, attention, and other scarce resources are unequally distributed…"
"[SIMs] are constituted through organized collective action."
"SIMs exist as historical entities for finite periods."
"SIMs can vary in intellectual aim and scope. Some problematize previously… underdiscussed topics… Others… seek to introduce entirely new theoretical perspectives on established terrain… Some SIMs distinguish themselves through new methods… Other SIMs aim to alter the boundaries of existing… intellectual fields…"

Next, you put forward some propositions about SIMs, which seem promising given the case studies you've seen, but are not the result of a comprehensive analysis of SIMs — merely a starting point:

"A SIM is more likely to emerge when high-status intellectual actors harbor complaints against what they understand to be the central intellectual tendencies of the day."
"SIMs are more likely to be successful when structural conditions provide access to key resources" (research funding, employment, access to rare equipment or data, intellectual prestige, etc.)
"The greater a SIM's access to [local sites at which SIM representatives can have sustained contact with potential recruits], the more likely it is to be successful."
"The success of a SIM is contingent upon the work done by movement participants to frame movement ideas in ways that resonate with the concerns of those who inhabit an intellectual field or fields."

My first question is this: what are the most significant pieces of follow-up work on your general theory of SIMs so far?

Scott Frickel: The article on SIMs that Neil Gross and I published back in 2005 has been well-received, for the most part. Citation counts on Google Scholar have risen steadily since then and so I'm encouraged by the continued interest. It seems that the article's central idea – that intellectual change is a broadly social phenomenon whose dynamics are in important ways similar to social movements – is resonating among sociologists and others.
The terrain that we mapped in developing our theory was intentionally quite broad, giving others lots of room to build on. And that seems to be what's happening. Rather than challenge our basic argument or framework, scholars' substantive engagements have tended to add elements to the theory or have sought to deepen theorization of certain existing elements. So for example, Jerry Jacobs (2013) extends the SIMs framework from specific disciplinary fields to the lines of connectivity between disciplines in seeking to better understand widespread enthusiasms for interdisciplinarity. Mikaila Arthur (2009) wants to extend the framework to better theorize the role of exogenous social movements in fomenting change within the academy. Tom Waidzunas (2013) picks up on our idea of an "intellectual opportunity structure" and, like Arthur, extends the concept's utility to the analysis of expert knowledge production beyond the academy. In his excellent new book, Why are Professors Liberal and Why do Conservatives Care? (Harvard, 2013), Neil Gross links our theory to the political leanings of the American professoriate. His idea is that SIMs can shape the political typing of entire fields – e.g. as more or less liberal or conservative. So, rather than arguing for an extended view of SIMs, Gross wants to recognize an extended view of the impacts of SIMs, which can affect academic fields singly or in combination with other competitor SIMs. Another study that I like very much is John Parker and Ed Hackett's (2012). analysis of how emotions shape intellectual processes in ways that drive the growth and development of SIMs. The emotional content of SIMs is something quite new for the theory, but which is consonant with lots of good work in social movement theory. Some of my own recent work builds from the SIMs project to offer a companion theory of 'shadow mobilization' to help explain expert interpenetration of social movements (Frickel et al. (2014)) So, in different ways, the project is chugging forward.

Luke: In your view, what is the expected eventual "payoff" from this kind of sociological research into social movements? Or to put it another way, who are the "consumers" (besides other sociologists) who at some point take certain results and apply them to solving real-world problems? (Your work on SIMs may be too young to have reached that stage yet, so that's why I'm asking about this kind of work more generally.)

Scott: So, if we think about the SIMs theory in applied terms as something like a recipe for building and sustaining expert collective action in ways that alter the intellectual and scientific landscape, I would say that our "product" is still under development. The various efforts to extend and refine the theory that I mentioned previously support that assessment. But to the extent that the theory could have broader resonance and more practical applications, it is interesting to think about who those consumers might be and what the theory might be good for.
I suppose the most obvious market for our theory would be academic experts who want to change the direction of research in their field. The theory might be useful to scientists who find comfort in the common but I think largely mistaken belief that science is driven by the best new ideas. Our theory begins from a very different assumption, namely that winning ideas are ideas whose champions strategically organize support for their projects by building networks and alliances to topple orthodoxies or defeat challengers. In this way of thinking, new ideas are necessary but insufficient explanations; knowledge advances not in spite of politics, but because of it.
The climate change debates in the US are a good case in point. Having good data and a theory to explain the data isn't enough. You also need organization and political savvy to anticipate and fend off the opposition. Climate change deniers figured this out first and built a movement that has proven tragically successful at blocking meaningful policy to reduce greenhouse gas emissions. Environmental sociologists Aaron McCright and Rile Dunlap (2003, 2010) have studied this phenomenon extensively as have historians of science Naomi Oreskes and Eric Conway (2010). In rising to meet the challenge of climate denialism, scientists have produced more and better data and worked harder to refine their theories, but they have also become better organized themselves, as has some sectors of the general public. This is precisely what social movement scholarship tells us about the dynamics between social movements and counter-movements. Our theory holds that this is also a regularly occurring dynamic in science. The climate change debates are not unique in this sense, only uniquely visible.

Luke: Which follow-up studies on your theory of SIMs would you most like to see in the next 10 years?

Scott: I imagine that detailed case studies such as those we've seen to date will continue to be an essential element of SIMs scholarship. But the case study method has limitations. By design it does not produce generalizable knowledge (if that is one's goal) because it prioritizes delivery of rich narrative detail of a single instance of the object of study. By themselves, case studies will tell us relatively little about broader political and institutional dynamics shaping SIM emergence and outcomes.
The studies that I am more interested in seeing done are those that develop a range of approaches for investigating broader questions relating to the historical forces and institutional environments that produce SIMs. Do SIMs emerge in similar fashion in different disciplinary domains or national contexts? Are certain kinds of academic institutions more nurturing of SIMs than others? Are there some areas of science, social science, and humanities that tend to not produce SIMs? These sorts of questions are best answered with studies that are explicitly comparative and I'd like to see research push in this direction.
I'd also like to see research that examines longer term historical patterns of SIM emergence and decline. Social movement scholars have developed the concept of "cycles of protest" to describe the historical ebb and flow of contentious political activity and social movement formation over longer periods of time, usually several decades or more (The original study on protest cycles is Tarrow 1989). They've used newspaper articles and other archival materials to collect data on protest events and then coded those accounts for various types of activity (e.g. strikes, boycotts, marches), numbers of people involved, how long the event lasted, whether violence was involved, how the authorities responded, and the like. These studies have shown that national-level protests tend to cluster in "waves" separated by periods of relative calm. Scholars have put forward various theories to explain these patterns, including the idea that initiating movements can have "spillover" effects on later-forming movements. This is the general pattern we see in the U.S. during the 1960s and 70s, with the civil rights movement feeding enthusiasms, building networks, and creating political opportunities that other movements could use to their advantage. What we now refer to as "The 60s" was, in part, a function of this major wave of political protest.
I'm very interested in whether and how SIMs pattern in similar ways. For example, can we find evidence that collective efforts to build new fields and research programs in the social sciences concentrate more in certain time periods than in others? There is prima facie evidence suggesting that "new" interdisciplinary social sciences emerged in the 1930s when we see fields like social psychology and criminology come about. We see clustering again in the 1970s, when universities begin creating degree programs for women's studies, African American studies, Chicano studies, and others. But these examples are far from comprehensive. The difficulty I'm facing is in working out an approach to data collection that allows us to systematically identify and code SIMs across disciplinary domains and over longer periods of time. Bibliometric data on journal publications and topic modeling techniques may hold some promise. This is the next big project that I'd like to tackle in this area of my research program – hopefully again in collaboration with Neil Gross. If you or any readers of this interview have thoughts about how to move this kind of study forward, I welcome your input.

Luke: Thanks, Scott!
The post Scott Frickel on intellectual movements appeared first on Machine Intelligence Research Institute.

Scott Frickel on intellectual movements


Nick Bostrom to speak about Superintelligence at UC Berkeley

MIRI has arranged for Nick Bostrom to discuss his new book — Superintelligence: Paths, Dangers, Strategies — on the UC Berkeley campus on September 12th.
Bostrom is the director of the Future of Humanity Institute at Oxford University, and is a frequent collaborator with MIRI researchers (e.g. see "The Ethics of Artificial Intelligence"). He is the author of some 200 publications, and is best known for his work in five areas:  (1) existential risk; (2) the simulation argument; (3) anthropics; (4) the impacts of future technology; and (5) the implications of consequentialism for global strategy. Earlier this year he was included on Prospect magazine's World Thinkers list, the youngest person in the top 15 from all fields and the highest-ranked analytic philosopher.
Bostrom will be introduced by UC Berkeley professor Stuart Russell, co-author of the world's leading AI textbook. Russell's blurb for Superintelligence reads:
Nick Bostrom makes a persuasive case that the future impact of AI is perhaps the most important issue the human race has ever faced. Instead of passively drifting, we need to steer a course. Superintelligence charts the submerged rocks of the future with unprecedented detail. It marks the beginning of a new era.
The talk will begin at 7pm at room 310 (Banatao Auditorium) in Sutardja Dai Hall (map) on the UC Berkeley campus.
If you live nearby, we hope to see you there! The room seats 150 people, on a first-come basis.
There will also be copies of Superintelligence available for purchase.

The post Nick Bostrom to speak about Superintelligence at UC Berkeley appeared first on Machine Intelligence Research Institute.


2014 Summer Matching Challenge!

Thanks to the generosity of several major donors,† every donation made to MIRI between now and August 15th, 2014 will be matched dollar-for-dollar, up to a total of $200,000!

$0

$50K

$100K

$150K

$200K

We have reached our matching total of $200,000!

116
Total Donors

Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund our research program.
Corporate matching and monthly giving pledges will count towards the total! Please email <EMAIL> if you intend on leveraging corporate matching (check here, to see if your employer will match your donation) or would like to pledge 6 months of monthly donations, so that we can properly account for your contributions towards the fundraiser.
(If you're unfamiliar with our mission, see: Why MIRI?)

 Donate Now

Accomplishments Since Our Winter 2013 Fundraiser Launched:

Hired 2 new Friendly AI researchers, Benja Fallenstein & Nate Soares. Since March, they've authored or co-authored 4 papers/reports, with several others in the works. Right now they're traveling, to present papers at the Vienna Summer of Logic, AAAI-14, and AGI-14.
5 new papers & book chapters: "Why We Need Friendly AI," "The errors, insights, and lessons of famous AI predictions," "Problems of self-reference…," "Program equilibrium…," and "The ethics of artificial intelligence."
11 new technical reports: 7 reports from the December 2013 workshop, "Botworld," "Loudness…," "Distributions allowing tiling…," and "Non-omniscience…"
New book: Smarter Than Us, published both as an e-book and a paperback.
Held one MIRI workshop and launched the MIRIx program, which currently supports 8 independently-organized Friendly AI discussion/research groups around the world.
New analyses: Robby's posts on naturalized induction, Luke's list of 70+ studies which could improve our picture of superintelligence strategy, "Exponential and non-exponential trends in information technology," "The world's distribution of computation," "How big is the field of artificial intelligence?," "Robust cooperation: A case study in Friendly AI research," "Is my view contrarian?," and "Can we really upload Johnny Depp's brain?"
Won $60,000+ in matching and prizes from sources that wouldn't have otherwise given to MIRI, via the Silicon Valley Gives fundraiser. (Thanks again, all you dedicated donors!)
49 new expert interviews, including interviews with Scott Aaronson (MIT), Max Tegmark (MIT), Kathleen Fisher (DARPA), Suresh Jagannathan (DARPA), André Platzer (CMU), Anil Nerode (Cornell), John Baez (UC Riverside), Jonathan Millen (MITRE), and Roger Schell.
4 transcribed conversations about MIRI strategy: 1, 2, 3, 4.
Published a thorough "2013 in review."

 Donate Now

Ongoing Activities You Can Help Support

We're writing an overview of the Friendly AI technical agenda (as we see it) so far.
We're currently developing and testing several tutorials on different pieces of the Friendly AI technical agenda (tiling agents, modal agents, etc.).
We're writing several more papers and reports.
We're growing the MIRIx program, largely to grow the pool of people we can plausibly hire as full-time FAI researchers in the next couple years.
We're planning, or helping to plan, multiple research workshops, including the May 2015 decision theory workshop at Cambridge University.
We're finishing the editing for a book version of Eliezer's Sequences.
We're helping to fund further SPARC activity, which provides education and skill-building to elite young math talent, and introduces them to ideas like effective altruism and global catastrophic risks.
We're continuing to discuss formal collaboration opportunities with UC Berkeley faculty and development staff.
We're helping Nick Bostrom promote his Superintelligence book in the U.S.
We're investigating opportunities for supporting Friendly AI research via federal funding sources such as the NSF.

Other projects are still being surveyed for likely cost and impact. See also our mid-2014 strategic plan.
We appreciate your support for our work! Donate now, and seize a better than usual opportunity to move our work forward. If you have questions about donating, please contact Malo Bourgon at <PHONE> or <EMAIL>.
† $200,000 of total matching funds has been provided by Jaan Tallinn, Edwin Evans, and Rick Schwall.
The post 2014 Summer Matching Challenge! appeared first on Machine Intelligence Research Institute.

2014 Summer Matching Challenge!


May 2015 decision theory conference at Cambridge University

MIRI, CSER, and the philosophy department at Cambridge University are co-organizing a decision theory conference titled Self-Prediction in Decision Theory and AI, to be held in the Faculty of Philosophy at the Cambridge University. The dates are May 13-19, 2015.
Huw Price and Arif Ahmed at Cambridge University are the lead organizers.
Confirmed speakers, in the order they are scheduled to speak, are:

Arif Ahmed (Cambridge)
Huw Price (Cambridge)
Julia Haas (WU St. Louis)
Wlodek Rabinowicz (Lund)
Kenny Easwaran (Texas A&M)
Preston Greene (NTU)
Joseph Halpern (Cornell)
Katie Steele (LSE)
Jenann Ismael (Arizona)
H. Orri Stefánsson (IFS)
Benja Fallenstein (MIRI)
Reuben Stern (U Wisconsin)
Nate Soares (MIRI)
Stuart Armstrong (Oxford)
Patrick LaVictoire (MIRI)
Catrin Campbell-Moore (MCMP)
James Joyce (U Michigan)
Alan Hájek (ANU)
Stuart Russell (Berkeley)
Vladimir Slepnev (Google)

(Updated May 17, 2015.)
The post May 2015 decision theory conference at Cambridge University appeared first on Machine Intelligence Research Institute.


MIRI's July 2014 newsletter

Research Updates

Two new reports: "Distributions allowing tiling of staged subjective EU maximizers" and "Non-omniscience, probabilistic inference, and metamathematics."
New analysis: Failures of an embodied intelligence.
Book chapter co-authored by Nick Bostrom (Oxford) and Eliezer Yudkowsky (MIRI) now published in the Cambridge Handbook of Artificial Intelligence.
2 new expert interviews: Roger Schell on long-term computer security research and Allan Friedman on cybersecurity and cyberwar.

News Updates

We've released our mid-2014 strategic plan update.
There are currently six active MIRIx groups around the world. If you're a mathematician, computer scientist, or formal philosopher, you may want to attend one of these groups, or apply for funding to run your own independently-organized MIRIx workshop!
Luke and Eliezer will be giving talks at the Effective Altruism Summit.
We are actively hiring for four positions: research fellow, science writer, office manager, and director of development. Salaries + benefits are competitive, visa assistance available if needed.

Other Updates

Luke has a personal blog now, which often discusses, or links to articles about, long-term AI outcomes.
Our Final Invention by James Barrat is now available in audiobook.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

Facebook  |  Twitter  |  Google+  |  Forward to a friend

You're receiving this because you subscribed to the MIRI newsletter.
unsubscribe from this list | update subscription preferences

The post MIRI's July 2014 newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s July 2014 newsletter


New report: "Non-omniscience, probabilistic inference, and metamathematics"

UC Berkeley student and MIRI research associate Paul Christiano has released a new report: "Non-omniscience, probabilistic inference, and metamathematics."
Abstract:
We suggest a tractable algorithm for assigning probabilities to sentences of first-order logic and updating those probabilities on the basis of observations. The core technical difficulty is relaxing the constraints of logical consistency in a way that is appropriate for bounded reasoners, without sacrificing the ability to make useful logical inferences or update correctly on evidence.
Using this framework, we discuss formalizations of some issues in the epistemology of mathematics. We show how mathematical theories can be understood as latent structure constraining physical observations, and consequently how realistic observations can provide evidence about abstract mathematical facts. We also discuss the relevance of these ideas to general intelligence.
What is the relation between this new report and Christiano et al.'s earlier "Definability of truth in probabilistic logic" report, discussed by John Baez here? In this new report, Paul aims to take a broader look at the interaction between probabilistic reasoning and epistemological issues, from an algorithmic perspective, before continuing to think about reflection and truth in particular.
The post New report: "Non-omniscience, probabilistic inference, and metamathematics" appeared first on Machine Intelligence Research Institute.

New report: “Non-omniscience, probabilistic inference, and metamathematics”


Roger Schell on long-term computer security research

 Roger R. Schell is a Professor of Engineering Practice at the University Of Southern California Viterbi School Of Engineering, and a member of the founding faculty for their Masters of Cyber Security degree program. He is internationally recognized for originating several key security design and evaluation techniques, and he holds patents in cryptography, authentication and trusted workstation. For more than decade he has been co-founder and an executive of Aesec Corporation, a start-up company providing verifiably secure platforms. Previously Prof. Schell was the Corporate Security Architect for Novell, and co-founder and vice president for Gemini Computers, Inc., where he directed development of their highly secure (what NSA called "Class A1") commercial product, the Gemini Multiprocessing Secure Operating System (GEMSOS). He was also the founding Deputy Director of NSA's National Computer Security Center. He has been referred to as the "father" of the Trusted Computer System Evaluation Criteria (the "Orange Book"). Prof. Schell is a retired USAF Colonel. He received a Ph.D. in Computer Science from the MIT, an M.S.E.E. from Washington State, and a B.S.E.E. from Montana State. The NIST and NSA have recognized him with the National Computer System Security Award. In 2012 he was inducted into the inaugural class of the National Cyber Security Hall of Fame.

Luke Muehlhauser: You have several decades of experience in high assurance computer security, much of it in the military and government, and you discussed your experience at length in this interview. One thing I'm curious about is this: do you know of cases where someone was worried about a computer security or safety challenge that wasn't imminent but maybe one or two decades away, and they decided to start doing research to prepare for that challenge anyway — e.g. perhaps because they expected the solution would require a decade or two of "serial" research and/or engineering work, with each piece building on the ones before it, and they wanted to be prepared to meet the challenge near when it arrived? Lampson's early identification of the "confinement problem" — a decade or two before anyone detected such an attack in the wild, to my knowledge — looks to me like it might be one such example, but maybe I'm misreading the history there.

Roger R. Schell: First to perhaps clarify the context of my responses, let me refine your introductory summary of my experience in high assurance computer security, which you characterize by saying "much of it in the military and government". My responses may be better understood by recognizing that although I am currently a Professor on the faculty of the University of Southern California, I spent the previous 28 years in industry in positions ranging from founder and executive in a few of information technology startups to corporate security architect and senior development manager for one of the largest software companies. This was substantially more time than the 22 years I spent in the military prior to that.
That said, your question about the reading of history does take me back to my military experience. You asked, about whether there was a case "where someone was worried about a computer security or safety challenge that wasn't imminent but maybe one or two decades away, and they decided to start doing research to prepare for that challenge anyway". From my perspective on that history the answer is a definite yes, as I noted in my 2001 paper on "Information security: science, pseudoscience, and flying pigs." That paper referred to a major instance of that as follows:
The failure of penetrate and patch to secure ADP systems in the late sixties helped stimulate the Ware Report [in 1970], which represented a codification of the state of understanding, which primarily was a realization of how difficult the problem was. This was one of those points where understanding of concepts came together enough to allow a significant step forward.
The Ware Report [in 1970] clearly identified the problem, but left it unresolved. That led to the Anderson Panel [in 1972], which defined the reference monitor concepts and conceived a program for evaluating and developing kernels.
That same paper notes that the "confinement problem" that you cited from Butler Lampson, was recognized years earlier and generally termed the "multilevel security" problem. Butler in 1973 made a significant contribution by provide a term and providing a bit of a taxonomy. But as noted in the above paper:
Development of early military systems concluded that some portions of the system require particularly strong security enforcement. Specifically, this enforcement was necessary to protect data whose loss would cause extremely grave damage to the nation. Systems that handled such data, and simultaneously included interfaces and users who were not authorized to access such data, came to be known as "multilevel".
It is pretty clear that stating the need to protect such data from loss is essentially equivalent to stating the need to solve what was later in 1973 referred to at the confinement problem. In fact by that time the "program" defined by Anderson was well underway, and it was as you put it "expected the solution would require a decade or two of 'serial' research and/or engineering work".
So, I agree that with the substance of your conclusion that the confinement problem is one such example, but I would refine your reading of history to note that it was "decided to start doing research to prepare for that challenge" quite a while before the multilevel security challenge was termed the "confinement problem". I think that refinement is consistent with the nice paper last year by Alex Crowell, et. al., entitled "The Confinement Problem: 40 Years Later", which cites the 1973 report by D. E. Bell and L. J. LaPadula specifically directed at a mathematical model to address the multilevel security problem.

Luke: If someone wanted to find additional examples of people working for a decade or two "in advance" on a difficult computer security or safety challenge, who else would you recommend they ask, and where would you recommend they look for examples?

Roger: I would not expect to find a lot of examples of people working for a decade or two "in advance" for a couple of reasons. Even the few cases of which I am aware are not particularly notable for having the significant results successfully brought to bear to meet the challenge when it arrived.
First, in the U.S. Business culture rewards for executives seem to significantly diminish with the amount of time until investments result in return. Therefore, it is quite difficult to get a business to make a significant and persistent commitment to have people working on a challenge a decade or two "in advance".
Second, in the U.S. government at the end of the cold war there seemed to be a significant loss of urgency in seriously addressing the problem of dealing with a witted adversary whose likely tool of choice is software subversion. The early efforts, such as the Anderson panel, were focused on high assurance security to address to what had more recently been termed the advanced persistent threat (APT). This was (and is) a genuinely "difficult" security challenge. For a number of years the government had given little attention to high assurance security.
In terms of past examples, the substantial Multics efforts over a number of years by Honeywell (initially General Electric) is one of the few. Decades in advance of widespread commercial need Multics addressed the challenge of creating a "computing utility" for which security was a central value proposition. Many decades later essentially this vision has been given the new name of "cloud computing" – unfortunately without significant attention to security. This early commercial investment was not particularly well-rewarded.
Intel provides a second example. The Multics innovations for security significantly influence Intel to make investments in their x.86 architecture in anticipation of demand for security. Their inclusion of Multics-like hardware segmentation and protection rings for security was not easy when the number of transistors a chip were scarce. Again, Intel did not get a good return on this investment. As professor Bill Caelli of Australia pointed out in his paper on trusted systems, the GEMSOS security kernel (for which I was the architect) was a rare example of an operating system actually using this powerful hardware support. This RTOS from a small business hardly constituted a major market win for the Intel investment in segmentation and protection rings.
In terms of other people to ask about this, I don't know many. Dr. David Bell is the one who has rather systematically looked at the evolution of solution for difficult computer security challenges, as reflected in his 2005 retrospective invited paper (plus the addendum) on his security model work at ACSAC several years ago. It is clear that he has rather carefully thought about these issues.

Luke: From your perspective, what are some of the most important contemporary avenues of research on high assurance security systems?
E.g. in software safety the experts will list things like formal verification, program synthesis, simplex architectures, verified libraries and compilers like in VST, progress in formal validation ala Cimatti et al. (2012), "clean-slate" high assurance projects HACMS, systems-based approaches like Leveson (2012), and tools that make high assurance methods easier to apply. There's some overlap between software safety and security, but what avenues of research would you name in a high assurance security context?

Roger: It seems to me that some of the most important contemporary avenues of research on high assurance security systems are those related to what over the years have been persistent "hard problems". In my 2001 ACSAC invited essay on Information security, I listed half a dozen "Remaining Hard Problems":

Verifying the absence of trap doors in hardware.
Verifying the absence of trap doors and other malicious software in development tools.
Covert timing channels.
Covert channels in end-to-end encryption systems.
Formal methods for corresponding source code to a formal specification, and object code to source.
Denial of service attacks.

It is not surprising, I suppose, that many of these were also in the high assurance security challenges identified more than 15 years earlier in the TCSEC (Orange Book) section on "Beyond Class (A1)". I am disappointed in our profession and its sponsors, and have to say that from my perspective, this is still a reasonable list of hard problems, and remains an important set of contemporary avenues of research. Unfortunately, relative to the importance of high assurance security, very little well-focused research effort is being directed to them. From this list, one area is receiving some attention, although often buried in larger, less significant, projects: recent research in hardware verification is producing valuable results – which was at the top of my list of hard problems.
In addition to the hard problems, there is an additional area that I consider at least as important in terms of the potential for dramatic positive impact. This is what David Bell in his couple of Looking Back papers I referenced earlier called for in terms of pursuit of enhanced high assurance "security in the form of crafting and sharing reference implementations of widely needed components." David called for "Producing security archetypes or reference implementations of common networking components". Such reference implementations are secure systems capabilities that technologists think they can figure out how to produce, but the devil is in the details, so only research that includes an implementation can confirm or deny that hypothesis. He provided his list of proposed reference implementations. I have more recently in various public presentations proposed my own list which includes the following:

MLS network attached storage (NAS)
High Assurance MLS Linux, Unix, *ix
Guards, filters, and other (CDSs)
Networked Windows (Thin Client)
Real-time exec (appliances)
Critical infrastructure platform
Identity mgt PKI (Quality Attribute)
MLS handheld network devices (e.g., PDA)
Confined financial apps (e.g., credit card)

In summary, my bottom line is that I believe that the most important contemporary avenues of research on high assurance security systems fall into theses two areas: (1) addressing the remaining hard problems for the future and (2) research projects that include creating reference implementations that create ways to leverage the successful research results of the past for practical secure systems.

Luke: You write "I am disappointed in our profession and its sponsors…" In retrospect, what seem to be the major obstacles to faster progress on the hard problems of computer security?
Two possible explanations that come to mind are: (1) Researchers and funders prefer easier research projects, because they have a higher chance of "success" even though the social value may be lower. And/or perhaps (2) hard problems tend to require long-term research efforts, and current institutions are less well-suited to executing long-term efforts.
But these are just two of my guesses; I'd like to hear your perspective on what the major obstacles seem to be.

Roger: Although not particularly insightful by itself, to start I think it is fair to say that the major reason for the lack of faster progress on the hard problems of computer security is the lack of substantial resources being applied in an informed way to address them. And that does quickly lead to your question of the major obstacles to doing so. It seems to me there are two tightly related aspects to that. First, there must be funders who are willing and able to sponsor such work. Second there must be motivated researchers interested in pursuing such research. Both are needed. Although practically speaking what researchers advocate and promote can significantly affect the funders, they are not likely to continue pursuing paths where funders seem independently disinterested (or even hostile).
I have already noted that in our culture rewards for executives seem to significantly diminish with the amount of time until investments result in return. Although that makes it difficult to get a commitment to long-term efforts, this is not unique to computer security, and various sustained research efforts are pursued notwithstanding that. Unfortunately, there is evidence that computer security continues to face some additional unique challenges in the availability of resources for addressing hard problems where the results can provide significantly higher assurance of security.
Among these challenges is the rather unusual characteristic of significant vested interests that may not really want high assurance security. I touched on some of this in my oral history interview you mentioned at the beginning of our exchange, and others have alluded to additional observations of this sort.
The comments that have been made related to potential financial vested interests include the following:

Cyber security is a multi-billion dollar industry, with much of that revenue generated as the result of the rampant flaws in consumer products from low-assurance cyber components. In my history interview I mentioned that that the Black Forest Group, which is a consortium of international Fortune 50 kind of companies, gave up their pursuit of a high assurance Public Key Infrastructure (PKI) when "they concluded that the vested interests against high assurance just made it impractical". That vested interest environment can discourage commercial sponsorship for working on the hard problems.
Beyond consumer products, I also noted a couple of anecdotal reports from the aerospace industry discouraging broadly applicable, reusable high assurance solutions, because it was a threat to a significant revenue stream they have from repeatedly addressing security on an ad hoc basis in similar contexts.
It seems the research community can have its own vested interest in having unsolved problems to work on. Several have commented on how what seems to be an almost systematic loss of corporate memory facilities resources for projects to reinvent the wheel. That does little to encourage researchers to give attention to the hard problems. I noted in my history interview reports that "people in the research community fought strongly against having The Orange Book be a standard because it dampened the interest in research".
The 1997 IEEE history paper by MacKenzie and Pottinger, reports the serious fragmentation of security research attention after moving away from a focused effort based on the Orange Book where "the path to achieving computer security appeared clear." Faster progress on the hard problems of computer security is unlikely in the absence of relatively focused research attention.

In addition there have been separate comments about the challenges related to government policy issues:

There has long been a tension between policy for security solution whose technology is closed restricted (e.g., classified, as in government cryptography) versus technology that is open and transparent (e.g., the Orange Book). This is discussed in some detail in the 1985 George Jelen (an NSA employee) Harvard University paper on "Information Security", and the policy issues persist to this day. Significant influence toward restricted results does not enhance the prospects for researching security solutions. The cancellation of major commercial work on the difficult problem a high secure virtual machine monitor (sorely needed today for cloud computing) is a real-world example from the past of the draconian effect of government restrictions on access to research results. Paul Karger's 1991 IEEE paper reported a major reason for that cancellation was that, "U.S. State Department export controls on operating systems at the B3 and Al levels are extremely onerous and would likely have interfered with many potential sale".
What has been called "the equities issue" reflects that intelligence gathering can benefit from exploiting vulnerabilities, and thus can create a powerful vested interest in there NOT being major progress in the hard problems of computer security. Bruce Schneier in his May 2008 blog argues that this is particularly a challenge when the same agency has major responsibly for both defensive solutions and exploiting vulnerabilities, e.g., he says "The equities issue has long been hotly debated inside the NSA."
Government policy can strongly favor their dominant control over security solutions, rather than encourage commercial development, especially for high assurance where it really matters. As mentioned by David Bell in his couple of Looking Back papers I cited earlier, a powerful monopolist ploy is to promise high assurance government endorsed solutions in the future to discourage applying in the near term resources supportive of commercial offerings. David reported that "Boeing's Dan Schnackenberg believed that NSA through MISSI put all the Class A1 vendors out of business," and I personally saw that kind of activity at that time, as well as more recently. There are other examples beyond MISSI with names like SAT, NetTop, SELinux, MILS and HAP, and like MISSI not one of them ever actually delivered verified high assurance, e.g., a Class A1 evaluation. As David Bell noted, such monopolistic government policy discourages commercial investment in making progress, including on hard problems.
Resources are always scarce, and it seems that current policy for addressing cyber security strongly emphasizes massive surveillance trying to find evidence of exploitation of weak systems. The meager resources applied to high assurance defenses (which is the focus of the hard problems) against a witted adversary pales in comparison to the billions being spent on surveillance. As I said about surveillance in my July 2012 keynote article for ERCIM, "this misplaced reliance stifles introduction of proven and mature technology that can dramatically reduce the cyber risks to privacy… an excuse for overreaching surveillance to capture and disseminate identifiable information without a willing and knowing grant of access."

In summary, I believe the primary reason we have not seen faster progress on the hard problems of computer security is the persistent decision not to apply significant resources. A major class of obstacles to applying resources are strong vested interest that are somewhat unique to computer security. There are not only commercial financial vested interests but also government policy vested interests that for years seem to have successfully created major obstacles.

Luke: Thanks, Roger!
The post Roger Schell on long-term computer security research appeared first on Machine Intelligence Research Institute.


New chapter in Cambridge Handbook of Artificial Intelligence

The Cambridge Handbook of Artificial Intelligence has been released. It contains a chapter co-authored by Nick Bostrom (Oxford) and Eliezer Yudkowsky (MIRI) called "The Ethics of Artificial Intelligence," available in PDF here.
The abstract reads:
The possibility of creating thinking machines raises a host of ethical issues. These questions relate both to ensuring that such machines do not harm humans and other morally relevant beings, and to the moral status of the machines themselves. The first section discusses issues that may arise in the near future of AI. The second section outlines challenges for ensuring that AI operates safely as it approaches humans in its intelligence. The third section outlines how we might assess whether, and in what circumstances, AIs themselves have moral status. In the fourth section, we consider how AIs might differ from humans in certain basic respects relevant to our ethical assessment of them. The final section addresses the issues of creating AIs more intelligent than human, and ensuring that they use their advanced intelligence for good rather than ill.
The post New chapter in Cambridge Handbook of Artificial Intelligence appeared first on Machine Intelligence Research Institute.


Our mid-2014 strategic plan

Summary
Events since MIRI's April 2013 strategic plan have increased my confidence that we are "headed in the right direction." During the rest of 2014 we will continue to:

Decrease our public outreach efforts, leaving most of that work to FHI at Oxford, CSER at Cambridge, FLI at MIT, Stuart Russell at UC Berkeley, and others (e.g. James Barrat).
Finish a few pending "strategic research" projects, then decrease our efforts on that front, again leaving most of that work to FHI, plus CSER and FLI if they hire researchers, plus some others.
Increase our investment in our Friendly AI (FAI) technical research agenda.

The reasons for continuing along this path remain largely the same, but I have more confidence in it now than I did before. This is because, since April 2013:

We produced much Friendly AI research progress on many different fronts, and do not remotely feel like we've exhausted the progress that could be made if we had more researchers, demonstrating that the FAI technical agenda is highly tractable.
FHI, CSER, and FLI have had substantial public outreach success, in part by leveraging their university affiliations and impressive advisory boards.
We've heard that as a result of this outreach success, and also because of Stuart Russell's discussions with researchers at AI conferences, AI researchers are beginning to ask, "Okay, this looks important, but what is the technical research agenda? What could my students and I do about it?" Basically, they want to see an FAI technical agenda, and MIRI is is developing that technical agenda already (see below).

In short, I think we tested and validated MIRI's new strategic focus, and now it is time to scale. Thus, our top goals for the next 6-12 months are to:

Produce more Friendly AI research.
Recruit more Friendly AI researchers.
Fundraise heavily to support those activities.

Our low-level tactics for achieving these high-level goals will probably change quickly as we try things and learn, but I'll sketch our current tactical plans below anyway.

1. More Friendly AI research
Our workshops have produced much FAI research progress, and they allowed us to recruit our two new FAI researchers of 2014, but we've learned that it's difficult for "newcomers" (people who haven't been following the research for a long time) to contribute to FAI research at our workshops. Newcomers need better tutorials and more time to think about the research problems before they can contribute much at the cutting edge (see the next section). Therefore our efforts toward novel research progress in 2014 will focus on:

Organizing research workshops attended mostly or solely by "veterans" (people who have been following the research for a long time), such as our May 2014 workshop.
Inviting individual researchers to visit MIRI for a few days at a time to work with us on very specific research problems with which they are already familiar.
Giving our staff FAI researchers time to make theoretical progress on their own and with each other.

2. Recruiting
Here I talk about "recruiting" rather than "creating a field," but in fact most of the planned activities below accomplish both ends simultaneously, because they find — and help to activate — new FAI researchers. Still, where there's a steep tradeoff between recruiting and helping to create a field, we focus on recruiting. FAI research is best done as a full-time career with minimal distractions, and right now MIRI is the only place offering such jobs.
We plan to try many different things to see what most helps for recruiting:

We plan to publish an overview of the FAI technical agenda as we see it so far. This should make it easier for potential FAI researchers to engage.
We plan to prepare, test, improve, and then deliver (in many different cities) a series of tutorials on different parts of the FAI technical agenda. Besides giving tutorial lectures, we may also organize one-day or two-day workshops that are just for tutorials and Q&A sessions rather than being aimed at making novel research progress.
We plan to release a book version of Yudkowsky's Less Wrong Sequences, in part because we've noticed that those who have contributed most to FAI research progress have read — and been influenced by — those writings, and this seems to help when they're doing FAI research.
We'll continue to help fund SPARC, the lead organizer of which is a frequent MIRI workshop participant (Paul Christiano).
We'll continue to fund independently-organized FAI workshops via our MIRIx program.
We may offer widely-advertised cash prizes for certain kinds of research progress, e.g. a winning decision algorithm submitted to a robust program equilibrium tournament, or a certain kind of solution to the Löbian obstacle.
We may advertise in venues such as Notices of the AMS (widely read by mathematicians) and Communications of the ACM (widely read by computer scientists).
We're in discussion with development staff at UC Berkeley about a variety of potential MIRI-Berkeley collaborations that could make it easier for researchers to collaborate heavily with MIRI from within a leading academic institution.

We've also made our job offer to FAI researchers more competitive.

3. Fundraising
To support our growing Friendly AI research program, we've set a "stretch" goal to raise $1.7 million in 2014.
To reach toward that goal, we participated in SV Gives on May 6th and raised ~$110,000, and also won ~$61,000 in matching and prizes from sources that otherwise wouldn't have donated to MIRI. Our thanks to everyone who donated!
As usual, we'll also run our major summer and winter fundraising drives, starting in July and December respectively.
This year we'll explore the possibility of corporate sponsorships, but we are not optimistic about that strategy, because MIRI's work has little near-term commercial relevance, and we've been counseled that corporate sponsors often require more in return from a sponsored organization than the funds are worth.
We're more optimistic about the potential returns from improving our donor stewardship and donor prospecting, which we've already begun, and we're seeking to hire a full-time Director of Development.
Donor prospecting is unlikely to result in new donors in 2014, but will be important for future fundraising years. This is also the case for our recent efforts to find and apply for grants from both private and public grantmakers. We don't expect to win much in grants in 2014, but we expect to learn in detail what else we need to do over the next couple years so that we can win large grants in the future, and thereby diversify our funding sources.
The post Our mid-2014 strategic plan appeared first on Machine Intelligence Research Institute.

Our mid-2014 strategic plan


New report: "Distributions allowing tiling of staged subjective EU maximizers"

MIRI has released a new technical report by Eliezer Yudkowsky, "Distributions allowing tiling of staged subjective EU maximizers," which summarizes some work done at MIRI's May 2014 workshop.
Abstract:
We consider expected utility maximizers making a staged series of sequential choices, and replacing themselves with successors on each time-step (to represent self-modification). We wanted to find conditions under which we could show that a staged expected utility maximizer would replace itself with another staged EU maximizer (representing stability of this decision criterion under self-modification). We analyzed one candidate condition and found that the "Optimizer's Curse" implied that maximization at each stage was not actually optimal. To avoid this, we generated an extremely artificial function η that should allow expected utility maximizers to tile. We're still looking for the exact necessary and sufficient condition.
The post New report: "Distributions allowing tiling of staged subjective EU maximizers" appeared first on Machine Intelligence Research Institute.

New report: “Distributions allowing tiling of staged subjective EU maximizers”


Allan Friedman on cybersecurity and cyberwar

MIRI recently interviewed Allan Friedman, co-author of Cybersecurity and Cyberwar: What Everyone Needs to Know.
We interviewed Dr. Friedman about cyberwar because the regulatory and social issues raised by the prospect of cyberwar may overlap substantially with those that will be raised by the prospect of advanced autonomous AI systems, such as those studied by MIRI.
Our GiveWell-style notes on this conversation are available in PDF format here.
The post Allan Friedman on cybersecurity and cyberwar appeared first on Machine Intelligence Research Institute.


MIRI's June 2014 Newsletter

Dear friends,

The SV Gives fundraiser was a big success for many organizations, and especially for MIRI. Thanks so much, everyone!
Research Updates

Two new papers: "Program equilibrium…" (accepted to the MIPC workshop at AAAI-14) and "Problems of self-reference…" (accepted for AGI-14).
First report from our May workshop: "Loudness: on priors over preference relations." (Other reports forthcoming.)
New analysis: Exponential and non-exponential trends in information technology.
9 new expert interviews, including e.g. Michael Fisher on verifying autonomous systems.

News Updates

Our MIRIx program wants to fund your independently-organized Friendly AI workshop.
We are actively hiring for four positions: research fellow, science writer, office manager, and director of development. Salaries + benefits are competitive, visa assistance available if needed.
Now available in paperback: Smarter Than Us: The Rise of Machine Intelligence.
Christof Koch and Stuart Russell on machine superintelligence.

Other Updates

The Future of Life Institute's inaugural talks and panel: The Future of Technology: Benefits and Risks (video).
A bit of humor: machine ethics on the Colbert Report.
New honors thesis by Danny Hintze compares four decision procedures, including Yudkowsky's TDT and Dai's UDT.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's June 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s June 2014 Newsletter