Robert Constable on correct-by-construction programming

Robert L. Constable heads the Nuprl research group in automated reasoning and software verification, and joined the Cornell faculty in 1968. He has supervised over forty PhD students in computer science, including the very first graduate of the CS department. He is known for his work connecting programs and mathematical proofs, which has led to new ways of automating the production of reliable software. He has written three books on this topic as well as numerous research articles. Professor Constable is a graduate of Princeton University where he worked with Alonzo Church, one of the pioneers of computer science.

Luke Muehlhauser: In some of your work, e.g. Bickford & Constable (2008), you discuss “correct-by-construction” and “secure-by-construction” methods for designing programs. Could you explain what such methods are, and why they are used?

Robert Constable:
Short Summary
The history of programming languages shows that types are valuable because compilers can check programs for type correctness and inform programmers whether or not their code exactly matches its type specification. Types describe requirements on programming tasks, and very rich type systems of the kind used in mathematics and increasingly in computing describe certain technical tasks completely as we’ll see below. The more expressive the type system, the more specifications help programmers understand their goals and the programs that achieve them. System designers can specify many requirements and computing tasks precisely with types and organize them into modules (or objects) that become a blueprint for analyzing and building a computing system.
When types are rich enough to completely specify a task, the methodology we just outlined becomes an example of correct by construction programming. This approach has been investigated for mathematical problems for decades and for programming problems since the late 60s and early 70s. For example, if we want to implement a correct by construction factorization based on the fundamental theorem of arithmetic (FTA), that every natural number greater than 1 can be factored uniquely into a product of primes, we can first express this task as a programming problem using types. The problem is then solved by coding a computable function factor with the appropriate type.  For programmers the type is probably just as clear as the FTA theorem. It is given below. It requires taking a natural number n greater than 1 as input and returning a list of pairs of numbers (p,e) where p is a prime and e is its exponent. We write the list as [(p1,e1), (p2,e2), …, (pn,en)] and require that n =p1e1 x … x pnen where pe means prime p to the power e, and the ei are all natural numbers, and x is multiplication (and is overloaded as the Cartesian product of two types). In addition the full FTA theorem requires that the factorization is unique. We discuss uniqueness later. For the factorization part, consider the example factor(24,750) = [(2,1),(3,2),(5,3),(11,1)]. This is correct since 24,750 = 2 x 9 x 125 x 11, where 53 is 125. In a rich type system the factorization piece of the fundamental theorem of arithmetic can be described completely as a programming task and the resulting program, factor, can be seen as a constructive proof accomplishing that factorization. A type checker verifies that the program has the correct type. The program factor is correct by construction because it is known that it has the type of the problem it was coded to solve.

Dependent Types
This paragraph is a bit technical and can be skipped – or used as a guide for coding factorization in a functional language with dependent types. Here is roughly how the above factorization task is described in the type systems of the programming languages of the Agda, Coq and Nuprl proof assistants – a bit more on them later. In English the type specification requires Nat, the type of the natural numbers, 0,1,2… and says this: given a natural number n larger than 1, compute a list of pairs of numbers, (p,e), where p is a prime number, and e is a natural number such that the input value n can be written as a product of all the prime numbers p on the list, each raised to the power e (exponent e) that is paired with it in the list lst. Symbolically this can be written as follows were N++ is the type (n:Nat where n>1). We also define the type Prime as (n:Nat where prime(n)) for a Boolean valued expression prime(n) which we code as a function from Nat to Booleans. Here is the FTA factorization type.
(n:N++) ->( lst:List[Prime x Nat] where n = prod(lst) ).
Note that the symbol x in (Prime x Nat) denotes the type constructor for ordered pairs of numbers. Its elements are ordered pairs (p,e) of the right types, p a Prime and e a Nat. Of course we need to define the product function, prod(lst), over a list lst of pairs in the right way. This is simple using functional programming in languages such as Agda, Coq, and Nuprl, in which we have the function type used above. We can assign this kind of programming problem to first year undergraduates, and they will solve it quite easily, especially if we give them a fast prime(n) procedure.
Another very nice way to write a compact solution is to use map and fold_left functions that are usually supplied in the library of list processing functions. That solution uses the well known map reduce paradigm taught in functional programming courses for years and widely used in industry. Solving with map and reduce is a good exercise since the reduce function fold_right is mathematical induction on lists defined as a specific very general recursive program.

Historical Perspective
The development of mainstream programming languages from Algol 60 to Java and OCaml shows that type systems have become progressively more expressive. For example, now it is standard that functions can be treated as objects, e.g. passed as inputs, produced as outputs, and created as explicit values as we did in the FTA example above. Type checking and type inference algorithms have kept up with this trend, although OCaml can’t yet check the type we created for FTA, Agda, Coq, Nuprl and other programming languages supported by proof assistants can (they all use some form of automated reasoning to help). Static type checking of programs defined using very rich types is a demonstrably effective technology for precisely specifying tasks, documenting code as it’s being written, rendering code more readable by others (including yourself when you revisit it several months later), finding errors, and for safely making changes. With higher-order types such as function types, more programming tasks can be precisely defined, and code can be generated that is completely correct by construction.
By 1984 researchers created an extreme point in this type checking technology in which the type systems are so rich that they can describe essentially any mathematical programming task in complete “formal” detail. Types also express assumptions on the data and the computing environment on which the specifications depend. These extremely rich type systems can essentially express any problem in mathematics. Indeed, Coq has been used to formally prove the Four Color Theorem as well as other important theorems in mathematics. The type checking by proof assistants often requires human intervention as well as sophisticated reasoning algorithms. The type systems we have mentioned are strong enough to also describe the uniqueness of the FTA factorization. They can do this by converting any factorization into the standard one produced by the function factor defined earlier. That function might be called normalize, and it does the job of converting any factorization into the standard one in which the primes come in order and are raised to the highest power possible.
Among the best known modern proof assistants are Agda, Coq, HOL, Isabelle, MetaPRL, Minlog, Nuprl, and PVS. Nuprl was the first of these, built in 1984, and the Coq assistant is written in OCaml and supported by INRIA through the French government. Coq is widely taught and used for programming. These proof assistants help programmers check extremely rich specifications using automated reasoning methods. The types are rich enough to guarantee that programs having the type are correct for the task specified. This is a dominant form of correct by construction programming and sometimes includes the technology of extracting a program from a proof that a specification is achievable; this is called using proofs as programs.

Research Goal
An overarching research goal is to design type systems and programming assistants to express any computational problem and provide mechanisms for type checking, perhaps with human aid and using automated reasoning tools, no matter how expressive the type system becomes.
Researchers in programming languages, formal methods, and logic are working on this goal worldwide. There are many references in the research literature citing progress toward this goal. The recent book by Adam Chlipala Certified Programming with Dependent Types, from MIT Press, 2013, provides an up to date view of the subject focused on the widely used Coq proof assistant. The 1991 book by Simon Thompson, Type Theory and Functional Programming, from Addison-Wesley, is one of the early comprehensive text books on the subject as it was just gaining momentum. Nuprl is the oldest proof assistant of this kind that is still active since 1984. It was described in the 1986 book Implementing Mathematics with The Nuprl Proof Development System. The Nuprl literature and an overview of the area circa 2006 can be found in the technical article “Innovations in computational type theory using Nuprl” from the Journal of Applied Logic, 4, 428-469, 2006. Also the author’s 2010 article on the “The Triumph of Types” at the 2010 celebration for Principia Mathematica in Cambridge, England recounts the 100 year old story of type theory and connects that to current research on type systems for correct by construction programming. It is available at www.nuprl.org.
When a type system can specify security properties, the checked programs are secure by construction. In both correct and secure by construction programming, the specifications include assumptions about the computing environment, say about the network topology. If these assumptions fail to hold, say because the network topology changed in an unexpected way, then the type specifications are no longer sufficient to guarantee correctness. So it is very important to document the assumptions on which the type specification depends.
For many problems these assumptions are stable, and in those cases, this correctness methodology is extremely effective. There are very good examples in the literature; at the web site www.nuprl.org we recently posted a simple example for the purpose of illustrating the method to newcomers. It is a complete formal proof that allowed us to build a correct by construction program to solve the “maximum segment sum” problem that is used to teach formal program development starting from type specifications.

Recent Work
The author and his colleagues at Cornell have recently used Nuprl to create a correct by construction version of the Multi-Paxos distributed consensus protocol that is being used in a database system. This required an implemented theory of distributed computing which is steadily evolving. We have made this protocol attack-tolerant, which is a form of security. Consensus protocols are critical to Cloud computing, and industry works very hard to build them correctly and make them secure as they are revised and improved. Researchers have created hundreds of correct by construction programs since 1984 and secure by construction programs as well. Worldwide many more are being built.
Modern proof assistants are steadily becoming more powerful and effective at correct and secure by construction programming because there is a research imperative and an economic incentive to use them. There is also a strong incentive to teach them, as the work on Software Foundations by Benjamin Pierce and his colleagues at the University of Pennsylvania demonstrates. Proof assistants are an addictive technology partly because they improve themselves the more they are used which steadily increases their appeal and value.

Luke: What do you think are the prospects for applying correct-by-construction and secure-by-construction approaches to methods commonly labeled “artificial intelligence”?

Robert: The proof assistants such as Agda, Coq, and Nuprl that support correct by construction programming are themselves examples of artificial intelligence in the sense that they use automated reasoning tools developed in AI. The earliest proof assistants such as the Boyer-Moore prover came out of the AI Department at Edinburgh University. Also one of the seminal books in the field, The Computer Modeling of Mathematical Reasoning, by Alan Bundy at Edinburgh was a landmark in AI, published in 1983 by Academic Press.
In due course these proof assistants will use other AI tools as well such as those that can translate formal proofs into natural language proofs. There has already been interesting progress on this front. For instance the article “Verbalization of High-Level Formal Proofs” by Holland-Minkley, Barzilay and me in the 16th National Conference on AI in 1999, 277 – 284 translates Nuprl proofs in number theory into natural language.
It is also the case that these systems are extended using each other. For example, right now my colleagues Vincent Rahli and Abhishek Anand are using the Coq prover to check that new rules they want to add to Nuprl are correct with respect to the semantic model of Nuprl’s constructive type theory that they painstakingly formalized in Coq. They have also checked certain rules using Agda. This activity is precisely what you asked about, and the answer is a clear yes.
When it comes to looking at machine learning algorithms on the other hand, the criteria for their success is more empirical. Do they actually give good performance? Machine learning algorithms were used by professors Regina Barzilay (MIT) and Lillian Lee (Cornell) to improve the performance of machine translation from Nuprl mathematics to natural language in their 2002 article on “bootstrapping lexical choice.” For this kind of work, it does not necessarily make sense to use correct by construction programming. On the other hand, one can imagine a scenario where a correct translation of the mathematics might be critical. In that case, we would probably not try to prove properties of the machine learning algorithm, but we would instead try to capture the meaning of the natural language version and compare that to the original mathematics.
The work of “understanding natural language mathematics” would benefit from correct by construction programming. Already there is excellent work being done to use constructive type theory to provide a semantic basis for natural language understanding. The book by Aarne Ranta, Type-theoretical grammar, Oxford, 1994 is an excellent example of this kind of work. It also happens to be one of the best introductions to constructive type theory.
Eventually machine learning algorithms are going to have a major impact on the effectiveness of proof assistants. That is because we can improve the ability of proof assistants to work on various subtasks on their own by teaching them how the expert humans handle these tasks. This kind of work will be an excellent example of both aspects of AI working together. Machine learning guides the machine to a proof, but the automated reasoning tools check that it is a correct proof.

Luke: What developments do you foresee, during the next 10-20 years, in proof assistants and related tools?

Robert: I’ve been working on proof assistants, programming logics, and constructive type theory for about forty years, and part of the job is to make predictions five to seven years out and try to attract funding to attain a feasible goal that will advance the field. To maintain credibility, you need to be right enough of the time. I’ve been more right than wrong on eight or so of these predictions. So I am confident about predicting five to seven years out, and I have ideas and hopes about ten years out, and then I have beliefs about the long term.

The Short Term
Over the next five years we will see more striking examples of correct by construction programming in building significant software and more examples of verifying software systems as in the work on the seL4 kernel, the SAFE software stack, features of Intel hardware, and important protocols for distributed computing, especially for cloud computing. This will be done because it is becoming cost effective to do it. I don’t see a major industrial effort even for the next ten years. This area will be a focus of continued academic and industrial research, and it will have incremental effects on certain key systems. However, it will remain too expensive for wide spread deployment in industry.
We will see formal tools being used to make both offensive and defensive weapons of cyber warfare more effective. One of the key lessons driving this is the fact that the stuxnet weapon, one of the best and most expensive ever created, was apparently defeated because it had a bug that led to its discovery. Another lesson is that the offensive weapons are improving all the time, and the defense must be more agile, not only building secure by construction “fortress like” systems, but thinking about systems as living responsive entities that learn how to adapt, recover, and maintain their core functionality.
In the realm of warfare, cost has rarely been the decisive factor, and formal methods tools are not outrageously costly compared to other required costs of maintaining a superior military. In this realm, unlike in building and maintaining ships, planes, and rockets and defenses against them, the cost lies in training and recruiting people. These people will have valuable skills for the private sector as well, and that sector will remain starved for well educated talent.
So overall, investing in advanced formal methods technology will be seen as cost effective. Proof assistants are the best tools we have in this sector, and we know how to make them significantly better by investing more. These tools include Agda, Coq, Nuprl, Minlog, and MetaPRL on the synthesis side, and ACL2, Isabelle-HOL, HOL-Light, HOL, and PVS on the verification side. All of them are good, each has important unique features. All have made significant contributions. All of them have ambitions for becoming more effective at what they do best. They will attract more funding, and new systems will emerge as well.

Medium Term
Most of the current proof assistants are also being used in education. It seems clear to me that the proof assistants that are well integrated with programming languages, such as Coq with OCaml and ACL2 with Lisp will prosper from being used in programming courses. Other proof assistants will move in this direction as well, surely Nuprl will do that. What happens in education will depend on the forces that bring programming languages in and out of fashion in universities. It seems clear to me that the Coq/OCaml system will have a chance of being widely adopted in university education, at least in Europe and the United States. It has already had a major impact in the Ivy League – Cornell, Harvard, Princeton, UPenn, and Yale are all invested as is MIT. I predict that within five years we will see this kind of programming education among the MOOCs, and we will see it move into select colleges and into top high schools as well.
All of the proof assistants have some special area of expertise. If there is sufficient funding, we will see them capitalize on this and advance their systems in various directions. Nuprl has been very effective in building correct by construction distributed protocols that are also attack tolerant. It is likely that this work will continue and result in building communication primitives into the programming language to match the theory of events already well developed formally. Coq has been used to build a verified C compiler, at UPenn and Harvard Coq is being used to verify the Breeze compiler for new hardware designed to support security features, and at MIT they are using Coq to verify more of the software stack with the Bedrock system. There are other applications of Coq too numerous to mention. We will see additional efforts to formally prove important theorems in mathematics, but it’s hard to predict which ones and by which systems. Current effortsto use proof assistants in homotopy theory will continue and will produce important mathematical insights. Proof assistants will also support the effort to build more certified algorithms along the lines of Kurt Mehlhorn’s impressive work.

Long Term
People who first encounter a proof assistant operated by a world class expert is likely to be stunned and shocked. This human/machine partnership can do things that seem impossible. They can solve certain kinds of mathematics problems completely precisely in real time, leaving a readable formal proof in Latex as the record of an hour’s worth of work. Whole books are being written this way as witnessed by the Software Foundations project at UPenn mentioned previously. I predict that we will see more books of readable formal mathematics and computing theory produced this way. Around them will emerge a technology for facilitating this writing and its inclusion in widely distributed educational materials. It might be possible to use proof assistants in grading homework from MOOCs. Moreover, the use of proof assistants will sooner or later reach the high schools.
I think we will also see new kinds of remote research collaborations mediated by proof assistants that are shared over the internet. We will see joint research teams around the world attacking certain kinds of theoretical problems and building whole software systems that are correct by construction. We already see glimpses of this in the Coq, Agda, and Nuprl community where the proof assistants share very similar and compatible type theories.
There could be a larger force at work for the very long term, a force of nature that has encoded mechanisms into the human gene pool that spread and preserve information, the information that defines our species. We might not be Homo sapiens after all, but Homo informatis. The wise bits have not always been so manifest, but the information bits advance just fine – at least so far. That is the aspect of our humanness for which we built the proof assistant partners. They are part of an ever expanding information eco system. There is a chance that proof assistants in a further evolved form will be seen by nature as part of us.

Luke: Thanks, Bob!
The post Robert Constable on correct-by-construction programming appeared first on Machine Intelligence Research Institute.


Armando Tacchella on Safety in Future AI Systems

Armando Tacchella is Associate Professor of Information Systems at the Faculty of Engineering, at the University of Genoa. He obtained his Ph.D in Electrical and Computer Engineering from the University of Genoa in 2001 and his “Laurea” (M.Sc equivalent) in Computer Engineering in 1997. His teaching activities include graduate courses in AI, formal languages, compilers, and machine learning as well as undergraduate courses in design and analysis of algorithms. His research interest are mainly in the field of AI, with a focus on systems and techniques for automated reasoning and machine learning, and applications to modeling, verification and monitoring of cyber-physical systems. His recent publications focus on improving the dependability of complex control architectures using formal methods, from the design stage till the operational stage of the system. He has published more than forty papers in international conferences and journals including AAAI, IJCAI, CAV, IJCAR, JAI, JAIR, IEEE-TCAD. In 2007 he was awarded by the Italian Association of Artificial Intelligence (AI*IA) the prize “Marco Somalvico” for the best young Italian researcher in AI.

Luke Muehlhauser: My summary of Transparency in Safety-Critical Systems was:
Black box testing can provide some confidence that a system will behave as intended, but if a system is built such that it is transparent to human inspection, then additional methods of reliability verification are available. Unfortunately, many of AI’s most useful methods are among its least transparent. Logic-based systems are typically more transparent than statistical methods, but statistical methods are more widely used. There are exceptions to this general rule, and some people are working to make statistical methods more transparent.
The last sentence applies to a 2009 paper you co-authored with Luca Pulina, in which you show formal guarantees about the behavior of a trained multi-layer perceptron (MLP). Could you explain roughly how that works, and what kind of guarantees you were able to prove?

Armando Tacchella: A trained MLP is just a mathematical function and, in principle, its properties could be verified in a logic that deals with real numbers and interpreted linear functions. However, useful MLPs, even the single hidden layer ones that we consider in our paper, use non-linear transcendental functions in their “activation” units (e.g., logistic or hyperbolic tangent). Any logic language that deals natively with such functions is bound to have a non-decidable decision problem — for a discussion see the introduction of this paper.
Our approach to establish safety of MLPs is thus based on an abstraction-refinement paradigm, whereby an “abstract” MLP is obtained from the “concrete” one using some over-approximation whose properties are decidable. This is a technique introduced by P. Cousot and R. Cousot some time ago and more recently exploited in the model checking community to attempt the verification of systems whose properties are not decidable.
If the abstract MLP is found  to satisfy the properties, then also the concrete one does. Otherwise, we obtain an “abstract” counterexample, i.e., a set of inputs which violates a property in the abstract MLP.
If the set of inputs is found to violate the property in the concrete MLP, then the concrete MLP is faulty, and must be fixed. On the other hand, if the concrete MLP works fine when supplied the abstract counterexample, such counterexample is “spurious”, meaning that it is an artifact of the abstraction which was too coarse to be useful.
The abstraction is thus “refined”, i.e., made less coarse, perhaps more expensive to check, and the process iterates, until either the MLP is proven safe, or the process exhausts available resources.
Clearly, abstraction-refinement, does not work around the initial undecidability issue, because an infinite chain of refinements due to spurious counterexamples might always exist. However, in practice, we are often able to establish the validity of a property with a reasonable number of refinements.
In our paper, we instantiate the approach above to the specific case of single hidden layer MLPs which are nevertheless universal approximators when subject to some reasonable conditions that we ensure. In particular:

We abstract MLPs to corresponding “interval functions”, i.e., if the MLP is y = f(x) then the abstract one is [y] = F([x]), where “F” is the interval function corresponding to “f”.
We consider a “safety property”, i.e., if [x] is contained in the input range of f, then [y] will be contained in some interval [l,h] with l < h (the “safety bound”).
We prove the property on F using a decision procedure for Linear Arithmetic over reals, e.g., a Satisfiability Modulo Theory (SMT) solver. If the property holds for F, i.e., for all [x] in the domain of f we have that F([x]) is contained in [l,h], then this is the case also for f.

If a counterexample is found, then there is some interval [x] such that F([x]) is not contained in [l,h]. We consider any x contained in [x], and we try the corresponding f(x) to check whether f(x) is in [l,h]. If one is found, then f is faulty and must be fixed. Otherwise, the counterexample was spurious, so we restart from step (1) with a finer grained refinement.
In the end, if the MLP f passes our test, we are guaranteed that whatever input it will receive, it will never exceed its “specifications” in term of minimum and maximum expected output values.

Luke: In what ways do you think that formal methods helped to “explain” the MLP, and make it more transparent to human programmers?

Armando: The answer to this question largely depends on what we mean by “explanation”.
If by “explanation” we mean that, using our verification methodology, the neural net user is now able to understand clearly which “reasoning rule” is implemented by which neuron (or set of neurons), then our method does not provide any further insight in that direction.
However, if we look for “behavioral explanation” then, the techniques proposed in our paper, helps in at least two ways:

the formal analysis of the original trained neural network enables the network user to establish whether its behaviors are in line with the specifications. When this is not the case, the user is given a (set of) counterexample(s) that will aid her in identifying the causes of misbehaviors. Indeed, by perusing the verification outputs, the user learns about the “points of failure” in the input/output space of the network, which, in turn, will enable him to improve his original design (e.g., by collecting more experimental evidence around those points)
since a neural network is not “classical” software — albeit from a formal point of view we treat it exactly as such — it might be the case that the user cannot readily “fix” the network using counterexamples. Our “repair” procedure can help to fix the network in an automated way by performing a controlled training using the result of “spurious” counterexamples. If the system to be learned by the network is available for experimentation, our repair procedure can also leverage the results of actual counterexamples (a form of “active learning”).

Technically, repair exposes the network to the danger of “overfitting” input data, i.e., the variance in the network performance is very high when considering several different datasets. However, we controlled for that possibility, and it never occurred in our experiments.

Luke: Before our interview, you wrote to me that:
I am quite skeptical that we will see any strongly autonomous robot around unless we are able to show (in a convincing way) that the robot is harmless for the environment…
Since I do not expect to be able to achieve safe strong autonomy with “correct-by-construction” and “proven in use” philosophies alone (the ones we use to build airplanes), I expect formal analysis to be even more important in robots than in any other implement.
Were you saying that, despite the tremendous economic and military incentives to build ever-more-capable machines without waiting for safety-related capabilities to catch up, companies and governments will in fact not release strongly autonomous programs and robots “into the real world” without first making sure that we have the kind of confidence about their safety that we currently require for (e.g.) autopilot software?

Armando: Engineers know very well the kind of pressure your are talking about and, in fact, if we consider any catalog of “horror stories” – see e.g. Software Horror Stories for more than 100 cases related only to SW engineering – we cannot rule out the possibility of something bad happening when autonomous robots will hit the market.
For robots endowed with a relatively weak autonomy, or with a relatively low probability of causing serious harm to the surrounding environment, incidents may also be part of a trade-off wherein robot makers are not willing to invest huge amounts of effort to eliminate them completely. In these cases, I expect traditional risk analysis, and engineering best practices to prevail over a major shift to precise verification methods.
For robots endowed with strong autonomy, and a non-negligible potential to cause serious harms to the environment, I believe there will be a strong deterrent towards souping-up traditional engineering practices and adopting them in domains that they cannot really handle.
To make the discussion concrete, let us assume that the autonomous car made by Google is going to be sold as a product like, say, a mobile phone. A shipping company can now buy a fleet of such cars, and send them driverless to delivery parcels at the doors of their customers. Technologically this is feasible today, but we will not see this happening any time soon. The reasons are several, but let me mention a few that I consider important:

Ethical. Do we really want to empower a machine to a level that it can autonomously decide, e.g., to run-down a pedestrian? Also an autopilot can fail and kill people, but it was clearly not the “will” of the autopilot to do so.
Technical. Can we ensure that without an operator ready to press “emergency stop” the car will never behave in an hazardous way? Planes are operated by very-well trained officials, who can always “go manual” if anything goes wrong in the autopilot.
Economical. Even if we had the technology to insure safety, would it be cost-effective? Right now, airplane makers can ensure, through standard engineering practices (e.g., redundancy, stress testing, preventive maintenance) that autopilots are cost effective within their operational requirements.
Legal. Whose is the liability if an autonomous car causes an accident? The issue becomes especially bothersome if there are more legal entities involved in the product, e.g., the car maker, the system integrator, the “AI” supplier, the operator (assuming on-field configuration is available).

Going back to my initial statement about the importance of formal methods in robots, I expect that a crisp understanding of the phenomena governing the “brains” of an autonomous robot and its interactions with its “body” and the surrounding environment, are of paramount importance if we wish to address any of the above issues in a satisfactory way.

Luke: Levitt (1999) argued that as intelligent systems become increasingly autonomous, and operate in increasingly non-deterministic, unknown environments, it will not be possible to model their environment and possible behaviors entirely with tools like control theory:
If robots are to put to more general uses, they will need to operate without human intervention, outdoors, on roads and in normal industrial and residential environments where unpredictable… events routinely occur. It will not be practical, or even safe, to halt robotic actions whenever the robot encounters an unexpected event or ambiguous [percept].
Currently, commercial robots determine their actions mostly by control-theoretic feedback. Control-theoretic algorithms require the possibilities of what can happen in the world be represented in models embodied in software programs that allow the robot to pre-determine an appropriate action response to any task-relevant occurrence of visual events. When robots are used in open, uncontrolled environments, it will not be possible to provide the robot with a priori models of all the objects and dynamical events that might occur.
In order to decide what actions to take in response to un-modeled, unexpected or ambiguously interpreted events in the world, robots will need to augment their processing beyond controlled feedback response, and engage in decision processes.
Thus, he argues, autonomous programs will eventually need to be decision-theoretic.
What do you think of that prediction? In general, what tools and methods do you think will be most relevant for achieving high safety assurances for the highly autonomous systems of the future?

Armando: I agree with Levitt when he says that “control theory” alone cannot deal with strongly autonomous robots. This is also what motivates the stream of research in Formal Methods dealing with Robotics and Automation (see, e.g., Workshop on Formal Methods for Robotics and Automation for a recent event on the subject).
However, it might be the case that “engaging in decision theoretic processes” while the robot is performing its tasks raises challenging — to say the least — computational issues. Indeed, it is well known that even simple factual, i.e., Boolean, reasoning may require exponential time to be carried out. Since we expect reality to be much more complex than simple 0/1 evidence, reasoning in every “logic” that the robot is endowed with is going to be computationally hard.
Even assuming the availability of powerful heuristics and/or approximations, the problem is still made tough by real-time requirements in the decision process. Indeed, the robot may not take forever to plan its course of actions, and in many cases high variances in response time are not acceptable as well.
At the same time, making sure that a strongly autonomous robot stays safe using off-line verification only, is also probably not going to happen. This is because of computational issues (combinatorial state-space explosion, undecidability of expressive logics to deal with time/space/probabilities), but also because, as in control theory, most of the situations that will arise while the robot is running are hard to frame.
In my opinion, what we might see is a combination of several approaches, including:

“Correct by construction” design and implementation
Formal Verification and testing at the design and implementation stage
Lightweight reasoning while the robot is executing

Reasoning during execution is probably going to be “self-reflective” by adopting e.g., efficient monitors, that can be automatically synthesized during the design process, and then deployed on the robot to ensure that certain safety conditions are met at all the times without the burden of “deep” logical reasoning.
Finally, actual robots will probably need to be “compliant” to humans, not only in their cognitive/reasoning abilities but also in their structure (see, e.g., recent development in “Soft Robotics”). A “compliant” structure will improve security and/or will make resoning about security of the robot easier, since some “intrinsic safety” can be achieved by the use of innovative materials and electronics. In other words, we should not worry just about a “safe mind”, but we must retain the principle that a safer “mind” requires also a safer “body”.

Luke: Sometimes I like to distinguish “bottom up” vs. “top down” approaches to the challenge of long-term AI safety — that is, to the challenge of making sure that we can continue to achieve confident safety guarantees for AI systems as they grow in autonomy and capability.
“Bottom up” approaches to long-term AI safety build incrementally upon current methods for achieving confident safety guarantees about today’s limited systems. “Top down” approaches begin by trying to imagine the capabilities of the AI systems we’ll have several decades from now, and use those speculations to guide research today.
I would describe your response to my previous question as a “bottom up” perspective on long-term AI safety, and I think it’s the kind of perspective most people working on AI safety would give. To illustrate what I mean by “top down” approaches to long-term AI safety, I’ll provide three examples: (1) Lampson (1979)‘s theoretical description of the “confinement problem,” which motivated the creation of the field of covert channel communication. (2) Orseau & Ring (2012)‘s work on the agent-environment boundary typically used as a simplifying assumption in reinforcement learning and other AI models, which I hope will inspire future research into more “naturalistic” AI models that will avoid problematic “Cartesian” mistakes in agent reasoning. (3) Weaver (2013)‘s discussion of paradoxes in rational agency, which I hope will motivate later research into how future self-modifying agent-based AIs can reason coherently about the consequences of modifying their own decision-making code.
Even if you aren’t familiar with these examples already, do you have an opinion on “top down” approaches to long-term AI safety in general, and how they should (or shouldn’t) inform today’s research efforts?

Armando: Indeed, I was not familiar with the references you cited, and I can see that they are definitely much more “visionary” than what I have suggested previously. Since my background is in Engineering, I must acknowledge a bias towards “bottom-up” views as you define them, or simply “reductionist” and “pragmatic” as I would define them.
Your citations reminded my of a paper by Weld and Etzioni. I often quote and cite this paper in my works because, in my opinion, it had the merit to raise the level of concern of the AI community about “unleashing” strongly autonomous agents in the physical world. The contents of the paper, albeit slightly outdated, still can provide some useful guidelines and vocabulary for current research on safe AI (at least they did it for me). While it is not the kind of paper I would conceive, I believe this kind of speculation is useful to prepare the ground for more tangible achievements.
On the other hand, if we are to chase the current push for stronger autonomy in robots acting in the physical world, then we should probably be very pragmatic about what we can do with current technology in order to ensure safety at all times. While it is possible that future advances in technology will make a “decision theoretic” agent possible within the resource constraints of a mobile robot, it is current technology which is enabling, e.g., autonomous cars to drive around. Therefore, if we want this kind of robots to become products in the next few years, I believe we should focus research on making them trustworthy by leveraging current scientific and technological developments.

Luke: Are there any other readings you’d recommend, besides Weld & Etzioni (1994), on the subject of long-term AI safety?

Armando: Weld and Etzioni returned on the subject of safe AI in their paper “Intelligent agents on the internet: Fact, fiction, and forecast”
A similar theme is touched upon by Eichmann in his “Ethical web agents“.
I have found out about the “Call to Arms” by Etzioni and Weld in a series of papers by Gordon, of which I would deem “Asimovian Adaptive Agents” as the most relevant.
I had just stumbled upon (but still did not have a chance to read thoroughly) a relatively recent book by Wilks “Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues” (see a review here). The book seems controversial at times, but it does contain many different views about artificial companions and it does raise some issues that we should probably keep well in mind when designing physically situated agents.
Finally, to have an idea about concrete projects going on in the field of robotics where safety is considered both a prominent value and an essential challenge, this site lists many interesting recent developments and ideas for further research.

Luke: Thanks, Armando!
The post Armando Tacchella on Safety in Future AI Systems appeared first on Machine Intelligence Research Institute.


Anders Sandberg on Space Colonization

Anders Sandberg works at the Future of Humanity Institute, a part of the Oxford Martin School and the Oxford University philosophy faculty. Anders’ research at the FHI centres on societal and ethical issues surrounding human enhancement, estimating the capabilities and underlying science of future technologies, and issues of global catastrophic risk. In particular he has worked on cognitive enhancement, whole brain emulation and risk model uncertainty. He is senior researcher for the FHI-Amlin Research Collaboration on Systemic Risk of Modelling, a unique industry collaboration investigating how insurance modelling contributes to or can mitigate systemic risks.
Anders has a background in computer science and neuroscience. He obtained his Ph.D in computational neuroscience from Stockholm University, Sweden, for work on neural network modelling of human memory. He is co-founder and writer for the think tank Eudoxa, and a regular participant in international public debates about emerging technology.

Luke Muehlhauser: In your paper with Stuart Armstrong, “Eternity in Six Hours,” you run through a variety of calculations based on known physics, and show that “Given certain technological assumptions, such as improved automation, the task of constructing Dyson spheres, designing replicating probes, and launching them at distant galaxies, become quite feasible. We extensively analyze the dynamics of such a project, including issues of deceleration and collision with particles in space.”
You frame the issue in terms of the Fermi paradox, but I’d like to ask about your paper from the perspective of “How hard would it be for an AGI-empowered, Earth-based civilization to colonize the stars?”
In section 6.3. you comment on the robustness of the result:
In the estimation of the authors, the assumptions on intergalactic dust and on the energy efficiency of the rockets represent the most vulnerable part of the whole design; small changes to these assumptions result in huge increases in energy and material required (though not to a scale unfeasible on cosmic timelines). If large particle dust density were an order of magnitude larger, reaching outside the local group would become problematic without shielding methods.
What about the density of intragalactic dust? Given your technological assumptions, do you think it would be fairly straightforward to colonize most of the Milky Way from Earth?

Anders Sandberg: I believe it would be straightforward given the technological assumptions we make, but to demonstrate it requires much more exploratory engineering footwork.
Intergalactic dust appears to be relatively thin, which is unsurprising: most of the contents of intergalactic space are primordial. There are contributions from galaxies, but they are relatively minor. Another way of seeing that the dust is thin is the absence of obscuration of remote galaxies: the total area along the line of sight is small.
This is unfortunately not true inside the galactic plane. At least in some directions dust is optically thick. The probability of a photon (or a fast spacecraft) hitting a dust particle along its trajectory is high. The size distribution of course matters: if it was all in very small particles the problem is different from than rarer pebbles. (If the total mass density is ρ kg/m3, and the average radius is r meters, then the number density N scales as ρ/r3 grains per cubic meter, while the surface area scales as r2N = ρ/r – finer dust can obscure more, down to the diffraction limit.) The actual size distribution for larger grains that would be a hazard is unfortunately not experimentally known – there is a fair deal known about smaller grains, but the ones we care about here are hard to observe.
The approach in our paper consists of just sending enough probes to have a good chance of one reaching the target without any collision. This has an effective distance cutoff rather dependent on the density of too large grains (d ≈ 1/σN, where σ is the probe cross section and N is the number density). One might argue that sending a lot of probes could work beyond that, but since the required number grows as exp(distance/d) this soon swamps any local resources. So clearly some shielding is necessary; this reduces N. (Getting a very small σ is also good up until the point where the length of the javelin-shaped probe becomes long enough that it starts to suffer lateral hits)
Now, showing that this shielding is achievable is where things get much more involved. This requires analysing the interaction between relativistic grains and a target. We have done an informal analysis of this (unpublished), and think it is possible to get the shielding up more than enough for interstellar travel if one can build atomically precise structures. This seems to be roughly within the capabilities assumed in the rest of the paper.
Another interesting issue is that in our galaxy most dust exists within the Thin Disk. This is a disk around 3000 lightyear thick (with a scale height of about 400 lightyears), which contain most of the gas and dust (plus star mass). Outside this is the thick disk, which is far clearer. Even interstellar hydrogen gas is a rather nasty proton beam from the perspective of a relativistic traveller, so it might actually be useful to go above the plane of the galaxy. This is a spiral galaxy problem: elliptical galaxies tend to have little dust or gas, so they are much easier to travel through.

Luke: What if we just talk about colonizing the local group or the local supercluster? How sensitive are those outcomes to the density of intergalactic space dust and the energy efficiency of rockets?

Anders: There is a nice trade-off between dangerous dust density and safe travel distance: twice the density of largish dust and your distance is halved. So in our paper we assumed intergalactic dust densities larger than a micron to be 10-35 gm/cm3, giving us a reach of 1,194 megaparsecs for using just a single probe. This is more than enough for the local supercluster (which is just 33 megaparsecs across). So even if dust was 36 times denser than we thought this would be achievable.
Note that in galaxies densities can be up to a few million times higher than outside. So there we should expect the distance of travel to be merely a thousand parsecs (if, as seems likely, there is also more grains in the interstellar medium per unit of density distances get even shorter – at least when flying through the dirtiest parts).
Going slower reduces the impact energy, and hence makes the dangerous dust density go down. This effect is however mainly felt for low velocity (<0.1 c) probes: as soon as you start going an appreciable clip of lightspeed you become sensitive to even rather small grains. However, for local colonization if you are not competing with anybody or trying to outrun cosmological expansion, a leisurely pace might be totally acceptable. The energy demands of course decrease a lot too: for 99% c travel you need to supply energy corresponding to 7 times the rest mass of the probe, and even 0.5 c requires 15% of the rest mass energy. 0.1 c is down to just 0.5%.
Some depends on how good rockets can be made. The rocket equation is my least favourite physics equation: it states that to get a velocity change you need to use an exponential amount of reaction mass (and it gets even worse for relativistic rockets). This is why our paper assumed external launchers to send off the probes: using rockets for launch too would have increased the energy and mass demands enormously. The total energy requirement difference between relatively inefficient fission rockets and efficient antimatter rockets is however less than an order of magnitude, since the efficiency boost is eaten up by a longer range and hence the need to send more probes.
Since the paper was written we have tinkered with laser-pushed rocket ideas: if one section of the probe continues ahead, sending back a very intense laser beam to slow the payload, this can be made more effective. There are some very interesting optical possibilities here if designs can be made atomically precise that would make slowing quite energy efficient. But in the end, I don’t think these considerations will provide many orders of magnitude difference.

Luke: As explained by your paper, your plan assumes the probes would be launched by “fixed launch systems” such as coilguns, quenchguns, laser propulsion, and particle beam propulsion, which are vastly more efficient than rockets. (Rockets are for deceleration near the end of a probe’s trip.) Could you give a brief description of these fixed launch systems? Which ones are you most confident will be technologically feasible?

Anders: Most people have heard about railguns, devices that uses the Lorenz force law to electromagnetically accelerate projectiles. These are based on an armature sliding along two current-bearing rails, pushed forward by the magnetic field through the loop of rails and armature. While able to generate tremendous acceleration they also tend to be *messy*, since the armature experiences friction with the rails, generating plasma discharges and erosion. Nevertheless, such systems exist today and can generate a few km/s velocity launches. While one can imagine scaling things up and use optimal materials, for relativistic launches railguns are likely entirely out.
In our paper we assumed some form of coil-guns: the projectile is never touching the launcher, and is accelerated by electromagnetic coils along the launch path. This has many practical benefits, like only needing currents for one or two coils at a time. The main limitations are switching speeds (which we assume can be handled by advanced technology, especially since the motion is very predictable), magnetic saturation of the projectile (which limits the acceleration per coil, but can be handled by using many more coils) and resistance in the coils. Using superconductors for the coils seems entirely reasonable for a large scale project with big resources. That also allows “quenchguns”, where energy is first stored in currents in superconducting coils that are then quenched, producing a very sharp field gradient that could propel the payload. According to the literature and experts we talked with these can in principle be made very efficient, converting most of the electrical energy into kinetic energy. They are based on well understood physics and can be seen as a modest extrapolation of current engineering to large scales.
We have not investigated particle beam propulsion, but it clearly has some potential. In a way a particle beam is a coilgun-accelerated stream of very small projectiles, and part of their momentum can be gathered by a projectile either by absorbing them (leading to heating and damage issues), or by using a mini-accelerator to “grip” them. There is not as much research on this as for electromagnetic propulsion or laser propulsion, but the physics seems sound. Whether it can be made efficient enough likely depends on whether the high efficiency of the particle accelerator can balance the inefficiencies of capturing their momentum at the receiving end. This is worth investigating further.
Laser propulsion has been experimentally demonstrated on small scales, and again is fairly well understood from a exploratory engineering standpoint. The main limitation on acceleration is heating of the launched object: ideally it would just reflect the beam back, but imperfect reflection means that the back part will heat up. Atomically precise manufacturing appears to allow surprisingly beam-resistant materials, even when based on current materials. Another problem for relativistic projectiles is that the beam will become redshifted as it speeds up, and that a normal beam will spread out as it gets further away from the source, reducing efficiency. It turns out these problems can to some extent be fixed by some more radical optics: it looks like it is physically possible to create free-floating “optical fibres” (actually a long line of very thin lenses) that allow the beam intensity to remain high even over significant distances, adapting as the effective wavelength changes. These ideas were developed more fully after the manuscript was finished, so we do not make use of them in it. But they suggest (if they work out when fully written up and simulated) that laser-launching (and laser-powered slowing) might be the most effective way for an advanced civilization to deliver relativistic payloads.

Luke: Now let’s talk about deceleration, where rockets may be needed. But before we get to rockets, let’s talk about more exotic solutions for deceleration: magnetic sails and Bussard ramjets. How might those work?

Anders: The Bussard ramjet is a classic idea for powering an interstellar starship: instead of lugging around a lot of reaction mass (the reason the rocket equation is so painful), why not gather interstellar gas and burn it in a fusion reactor? Unfortunately this requires a pretty high speed in order to get an appreciable mass inflow, and constructing a ramscoop that sucks up enough mass is daunting. It would probably have to be a vast electromagnetic funnel, requiring significant power to generate (and then some more power to compress the input for fusion). Most calculations in the literature end up with the depressing conclusion that it would not be feasible unless the scoop and fusion reaction were very effective, and the exhaust speed very high.
However, one might of course turn the thing around and instead use the funnel to slow the ship against the interstellar medium. Dana Andrews and Robert Zubrin suggested using a magnetic sail, where the field from a superconducting loop deflects charged particles and hence experiences drag. This might be quite effective for slowing relativistic probes; they get a result that implies that a 99% c probe could be slowed to typical interplanetary speeds in about two centuries.
For slowing in intergalactic space where gas is thin ramjets or magnetic sails are of course not very effective until the probe has come very close to the destination (however, if it takes centuries to brake, the braking distance is still merely a few hundred light-years). The lack of reaction mass is very enticing. In our paper we did not explore these possibilities since we wanted a rather conservative estimate, but it looks likely that magnetic sails are worth a serious look in further modelling.

Luke: Why do you hate the relativistic rocket equation? How did you come up with your estimates for the energy efficiency of future rocket designs that could be used for deceleration?

Anders: The rocket equation states that the amount of reaction mass you need to expel grows exponentially with the desired ending velocity – this is why standard rockets are mostly fuel and the payload small. You can get better performance by expelling the reaction mass faster, up to the limit of light-speed (which would require a rocket that expels photons).
The relativistic rocket equation is even worse: as you speed up a lot of the energy turns into mass of the rocket rather than velocity. No amount of energy will get you beyond lightspeed, and as you approach it the acceleration decreases (as seen by an outside observer; somebody onboard the rocket will not notice anything). So the overall energy efficiency becomes worse and worse; unless you really want to get to your destination fast it might not be worth it.
The energy estimates were taken from the literature; see our citations.
This is one area where engineering realities might make designs less efficient, and it does have a big effect: a factor of 50% lower efficiency will square the initial mass to payload ratio. Ouch. Still, given the enormous amount of energy, mass and time available the end result is not too different: very large number of probes can be sent by the resources of a single solar system over astronomically insignificant timescales. It is practically irrelevant if it takes 15 years instead of 6 hours to launch the probes. Inefficiencies however start to bite more sharply the larger the replicator is: in the 500 ton case a 45% less efficient fusion reactor means a complete launch of 1.52 billion probes takes a billion years. So the lesson is clearly that boosting efficiency and/or producing smaller payloads have a big effect. If someone were to argue against our analysis they should likely focus here, looking for fundamental limits of efficiency or credible payload sizes. However, as discussed above, there might be alternatives like magnetic sails that vastly improve efficiency too.

Luke: Which few “next analyses” would be most desirable, from your perspective, for helping to answer the question of how likely it is that an AGI-empowered Earth-based civilization could colonize the galaxy, the supercluster, or the observable universe?

Anders: Much of the work can be done using exploratory engineering: using known physics, can we make designs that perform well enough to fit into the overall mission profile? They do not have to be optimal to prove that it could be done by an advanced civilisation (indeed, when thinking about the Dyson shell material requirements we considered a “steampunk” approach consisting of iron mirrors focusing sunlight on boilers with water or gas-filled turbines in order to check that even if Mercury or similar materials sources lacked elements for big photovoltaics large-scale energy production was still possible). The more detail, the more convincing, but effort should mainly be directed at the big bottlenecks of efficiency or where our understanding of engineering is a bit weak. (More stars = higher priority.)
★ Obviously better designs for self-replicating installations with materials closure in generic space environments are nice: the classic NASA lunar factory study is ancient by now. This also helps understand constraints on how quickly industrialisation of a system could be done, whether there are large first-mover advantages in the solar system that might pose arms race temptations, and some firmer numbers of seed sizes and replication times.
★ Understanding the AI needs for space mining and engineering would be useful. Full AGI is presumably by assumption able to run the infrastructure we require, but even lower order intelligence might be adequate: many animals construct elaborate constructions adaptively or navigate in complex environments without much general intelligence. This also helps estimate the importance of AI and AGI for human space industry.
★★ Relativistic mass drivers look feasible but cumbersome. We think investigating the limits and possibilities of laser-powered launching might be more productive.
★★★ We need a better understanding of the large particle distribution in the interstellar and intergalactic medium. Much of probe design hinges on whether to attempt a shielded probe or just send a large number of expendable probes.
★★ Slowing down using magnetic sails or lasers appear promising, but requires a good power source that can last for a very long time for intergalactic trips, presumably induced fission, fusion or antimatter. How do long-term storable energy sources scale? What are their minimum mass?
★ Automated navigation and colonization requires automated astronomy. We assumed the probes knew where they were going, but it would be interesting to analyse automated colonisation planning. Large orbiting infrastructures can likely act as very powerful telescopes.
★ From a Fermi question perspective it would be useful to investigate more the properties of spread in the younger universe: is the shorter distances outweighed by higher densities?
★★ From a strategic perspective it would be useful to investigate whether this type of replicating probes can be successfully prevented from invading an occupied galaxy or not. Much of long-term strategy may depend on whether invaders or defenders have the advantage, including whether very large singleton systems are possible or whether significant amount of resources have to be spent on defence.

Luke: Thanks, Anders!
The post Anders Sandberg on Space Colonization appeared first on Machine Intelligence Research Institute.

Anders Sandberg on Space Colonization


The world’s distribution of computation (initial findings)

What is the world’s current distribution of computation, and what will it be in the future?
This question is relevant to several issues in AGI safety strategy. To name just three examples:

If a large government or corporation wanted to quickly and massively upgrade its computing capacity so as to make a push for AGI or WBE, how quickly could it do so?
If a government thought that AGI or WBE posed a national security threat or global risk, how much computation could it restrict, how quickly?
How much extra computing is “immediately” available to a successful botnet or government, simply by running existing computers near 100% capacity rather than at current capacity?

To investigate these questions, MIRI recently contracted Vipul Naik to gather data on the world’s current distribution of computation, including current trends. This blog post summarizes our initial findings by briefly responding to a few questions. Naik’s complete research notes are available here (22 pages). This work is meant to provide a “quick and dirty” launchpad for future, more thorough research into the topic.

Q: How much of the world’s computation is in high-performance computing clusters vs. normal clusters vs. desktop computers vs. other sources?
A: Computation is split between application-specific integrated circuits (ASICs) and general purpose computing: According to Hilbert & Lopez (2011a, 2012a, 2012b), the fraction of computation done by general-purpose computing declined from 40% in 1986 to 3% in 2007. The trend line suggests further decline.
Within general-purpose computing, the split for the year 2007 is as follows:

For installed capacity: 66% PCs (incl. laptops), 25% videogame consoles, 6% mobile phones/PDAs, 3% servers and mainframes, 0.03% supercomputers, 0.3% pocket calculators.
For effective gross capacity: 52% PCs, 20% videogame consoles, 13% mobile phones/PDAs, 11% servers and mainframes, 4% supercomputers, 0% pocket calculators.

For more detailed data, see Section 2.2 of Naik’s research notes and Section E of Hilbert & Lopez (2011b).

Q: What is it being used for, by whom, where?
A: See the answer above, plus Section 3 of Naik’s research notes.

Q: How much capacity is added per year?
A: Growth rates and doubling periods are as follows, using data 1986-2007:

General-purpose computing capacity: growth rate 58% per annum, doubling period 18 months  (see Section 2.2 of Naik’s research notes).
Communication: growth rate 28% per annum, doubling period 34 months (see Section 2.3 of Naik’s research notes).
Storage: growth rate 23% per annum, doubling period 40 months (see Section 2.4 of Naik’s research notes).
Application-specific computing: growth rate 83% per annum, doubling time 14 months (see Section 2.2 of Naik’s research notes).

Breakdown of data by time periods is available in Hilbert & Lopez (2011a), and the most important quotes are included in the relevant sections of Naik’s research notes.

Q: How quickly could capacity be scaled up (in the short or medium term) if demand for computing increased?
The semiconductor industry is quite responsive to changes in demand, and catches up with book-to-bill ratios as large as 1.4 within 6 months (see Section 3.2 of Naik’s research notes). In addition, the fact that Litecoin, an allegedly ASIC-resistant substitute for Bitcoin, already has ASICs about to be shipped (within two years of launch) also suggests relatively rapid turnaround given large enough economic incentives. In the case of high-frequency trading (HFT), huge investments in nanosecond computing and shaving off milliseconds from Chicago-New York and New York-London cables also suggests quick responsiveness to large incentives.

Q: How much computation would we expect to be available from custom hardware (FPGA’s/ASIC’s and their future analogs)?
A: An increasing fraction (note decline in general-purpose computing’s share from 40% in 1986 to 3% in 2007). However, stuff like ASICs and FPGAs can’t really be repurposed for other use, so existing ones don’t help that much with new tasks.

Q: What is the state of standard conventions for reporting data on such trends?
A: The work of Hilbert, Lopez, and others may eventually lead to uniform conventions for reporting and communicating the data, which would allow for a more informed discussion of these. However, Martin Hilbert in particular is skeptical of standardization in the near future, although he believes it possible in principle, see Hilbert (2012) for more. On the other hand, Dienes (2012) argues for standardization.
The post The world’s distribution of computation (initial findings) appeared first on Machine Intelligence Research Institute.


Nik Weaver on Paradoxes of Rational Agency

Nik Weaver is a professor of mathematics at Washington University in St. Louis. He did his graduate work at Harvard and Berkeley and received his Ph.D. in 1994. His main interests are functional analysis, quantization, and the foundations of mathematics. He is best known for his work on independence results in C*-algebra and his role in the recent solution of the Kadison-Singer problem. His most recent book is Forcing for Mathematicians.

Luke Muehlhauser: In Weaver (2013) you discuss some paradoxes of rational agency. Can you explain roughly what these “paradoxes” are, for someone who might not be all that familiar with provability logic?

Nik Weaver: Sure. First of all, these are “paradoxes” in the sense of being highly counterintuitive — they’re not outright contradictions.
They all relate to the basic Löbian difficulty that if you reason within a fixed axiomatic system S, and you know that some statement A is provable within S, you’re generally not able to deduce that A is true. This may be an inference that you and I would be willing to make, but if you try to build it into a formal system then the system becomes inconsistent. So, for a rational agent who reasons within a specified axiomatic system, knowing that a proof exists is not as good as actually having a proof.
This leads to some very frustrating consequences. Let’s say I want to build a spaceship, but first I need to be sure that it’s not going to blow up. I have an idea about how to prove this, but it’s extremely tedious, so I write a program to work out the details of the proof and verify that it’s correct. The good news is that when I run the program, it informs me that it was able to fill in the details successfully. The bad news is that I now know that there is a proof that the ship won’t blow up, but I still don’t know that the ship won’t blow up! I’m going to have to check the proof myself, line by line. It’s a complete waste of time, because I know that the program functions correctly (we can assume I’ve proven this), so I know that the line by line verification is going to check out, but I still have to do it.
You may say that I have been programmed badly. Whoever wrote my source code ought to have allowed me to accept “there is a proof that the ship won’t blow up” as sufficient justification for building the ship. This can be a general rule: for any statement A, let “there exists a proof of A” license all the actions that A licenses. We’re not contradicting Löb’s theorem — we still can’t deduce A from knowing there is a proof of A — but we’re finessing it by stipulating that knowing there’s a proof of A is good enough. But there are still problems. Imagine that I can prove that if the Riemann hypothesis is true, then the ship won’t blow up, and if it’s false then there exists a proof that the ship won’t blow up. Then I’m in a situation where I know that either A is true or there is a proof that A is true, but I don’t know which one. So even with the more liberal licensing condition, I still can’t build my lovely spaceship.

Luke: Section 2 of the paper develops an approach to solving this kind of problem via the notion of warranted assertibility. How does that work?

Nik: “Warranted assertibility” is a philosophical term that, in mathematical settings, refers to statements which can be asserted with perfect rational justification. My idea was that this concept could help with these Löbian type problems because it is nicer in some ways than formal provability. For instance, in any consistent axiomatic system there are statements A(n) with the property that for each value of n we can prove A(n), yet there is no single proof of the statement “for all n, A(n)”. However, it is always the case that if each A(n) is warranted then “for all n, A(n)” is also warranted.
So I introduce a predicate Box(A) to express “A is warranted”, and given a suitable axiomatization (which is a little subtle) I show that we can always infer Box(A) from knowing that there is a proof of A. We still can’t infer A — Löb’s theorem is not violated — but, along the lines I suggested earlier, we can program our agents to accept Box(A) as licensing the same actions that A licenses. Thus, it suffices to prove that the statement “the ship won’t blow up” is warranted. If you go this route then all the paradoxes seem to disappear. If an agent is able to reason about warranted assertibility, then it gets to perform all the actions it intuitively ought to be allowed to perform.

Luke: Later in the paper, you write:
[Yudkowsky & Herreshoff (2013)] involves a rational agent who is seeking… general permission to delegate the performance of actions to a second agent… The same issues appear in the case of an intelligent machine that is considering modifying its own source code (in order to make itself more intelligent, say). Before doing this it would want to be sure that its post-modiﬁcation state will reason correctly, i.e., any theorem it proves after the modiﬁcation should actually be true. This runs into the familiar Löbian difficulty that the agent is not even able to affirm the soundness of its pre-modiﬁcation reasoning.
… In Section 4 of [Yudkowsky & Herreshoff (2013)] two constructions are presented of an infinite sequence of independently acting agents, each of whom can give a provable justification for activating the next one, yet none of whose deductive power falls below an initially prescribed level. The constructions are clever but they have a nonstandard flavor. Probably this is unavoidable, unless the problem description were to be altered in some fundamental way. In the remainder of this section we present another solution which uses nonstandard assertibility reasoning.
Can you describe the solution you go on to present — again, in terms that someone lacking a background in provability logic might be able to understand, at least in an intuitive way?

Nik: I will try! First, in regard to my comment that only nonstandard solutions seem possible, I should mention that there is now a theorem due to Herreshoff, Fallenstein, and Armstrong that makes this idea precise. The point is that it’s not clear I should be allowed to build a duplicate copy of myself and delegate my goals to it, for fear that it would then do the same thing, and on ad infinitum (the “procrastination problem”).
Now, going back to my spaceship example, let B be the statement that the ship won’t blow up. A basic feature of assertibility is that we have a general law that A implies Box(A), but we don’t have the converse. So Box(B) is “weaker” than B, and my proposal was that we should allow our agents to accept Box(B) to license the same action (building the spaceship) that B does. This helps with Löbian issues because the statement that B is provable doesn’t imply B, but it does imply Box(B).
But if Box(B) is good enough, why not accept the even weaker statement Box(Box(B))? Or Box(Box(Box(B)))? Let me write Box^k(B) for the general expression with k boxes. The idea about delegating tasks to successors is that if I am willing to accept Box^9(B), say, then I should be willing to build a successor who can’t accept Box^9(B) but can accept Box^8(B). I can reason that if my successor proves Box^8(B), then Box^8(B) is provable, and therefore Box^9(B) is true, which is good enough for me. Then my successor should be happy with a subsequent model that is required to prove Box^7(B), and so on. So we can create finite chains in this way, without doing anything nonstandard.
The way to get infinite chains is to introduce a nonstandard “infinite” natural number kappa, and start with an initial agent who is required to verify Box^kappa(B). Then it can build a successor who is required to verify Box^{kappa – 1}(B), and so on. All the agents in the resulting sequence have the same formal strength because a simple transposition (replacing kappa with kappa + 1) turns each agent into its predecessor, without affecting what they’re able to prove.

Luke: Now let me ask about research of this type in general. What’s your model for how this kind of research might be useful? It’s pretty distant from current methods for constructing actual software agents.

Nik: Yes, it’s very theoretical, no question. One model is pure mathematics, which is actually my background. In pure math, most of the time we’re working on questions that don’t have any immediate practical applications. I think experience has shown that this kind of work occasionally becomes very important, while most of it does not, but it’s hard to predict which directions are going to pan out.
Or, you know, maybe we’re all living inside a simulation like Nick Bostrom says. In which case, the fact that we’re trying to figure out how to help machines make themselves smarter may be a clue as to the nature of that simulation.

Luke: What do you think are some “promising avenues” for making additional progress on reasoning our way through these “paradoxes” of rational agency?

Nik: I don’t know. If you’re talking about the Löbian obstacle, I feel that something essential is missing from our current problem description. Among the directions that are available now my preference is for Benja Fallenstein’s parametric polymorphism. But I also think we should be looking at ways of modifying the problem description.

Luke: What is the origin of your interest in paradoxes of rational agency?

Nik: I’ve been thinking about issues surrounding truth and circularity for around five years. (I just checked: my earliest paper on the subject was posted on the Mathematics arXiv in May 2009.) This was purely theoretical work, coming from an interest in the foundations of mathematics and logic.
My involvement in paradoxes of rational agency was sparked by Eliezer Yudkowsky and Marcello Herreshoff’s paper on the Lobian obstacle. In fact I think I got the term “paradoxes of rational agency” from that paper. When I read it I had the immediate reaction that my ideas about assertibility should be relevant, so that’s how I got interested.

Luke: Thanks, Nik!
The post Nik Weaver on Paradoxes of Rational Agency appeared first on Machine Intelligence Research Institute.


MIRI’s May 2014 Workshop

From May 3–11, MIRI will host its 7th Workshop of Logic, Probability, and Reflection. This workshop will focus on decision theory and tiling agents.
The participants — all veterans of past workshops — are:

Mihály Barasz (Google)
Paul Christiano (UC Berkeley)
Benja Fallenstein (Bristol U)
Marcello Herreshoff (Google)
Patrick LaVictoire (U Wisconsin)
Nate Soares (Google)
Nisan Stiennon (Stanford)
Qiaochu Yuan (UC Berkeley)
Eliezer Yudkowsky (MIRI)

If you have a strong mathematics background and might like to attend a future workshop, apply today! Even if there are no upcoming workshops that fit your schedule, please still apply, so that we can notify you of other workshops (long before they are announced publicly).
The post MIRI’s May 2014 Workshop appeared first on Machine Intelligence Research Institute.

MIRI’s May 2014 Workshop


Conversation with Holden Karnofsky about Future-Oriented Philanthropy

Recently, Eliezer and I had an email conversation with Holden Karnofsky to discuss future-oriented philanthropy, including MIRI. The participants were:

Eliezer Yudkowsky (research fellow at MIRI)
Luke Muehlhauser (executive director at MIRI)
Holden Karnofsky (co-CEO at GiveWell)

We then edited the email conversation into a streamlined conversation, available here.
See also four previous conversations between MIRI and Holden Karnofsky: on existential risk, on MIRI strategy, on transparent research analyses, and on flow-through effects.
The post Conversation with Holden Karnofsky about Future-Oriented Philanthropy appeared first on Machine Intelligence Research Institute.


John Baez on Research Tactics

John Baez is a professor of mathematics at U.C. Riverside. Until recently he worked on higher category theory and quantum gravity. His internet column This Week’s Finds dates back to to 1993 and is sometimes called the world’s first blog. In 2010, concerned about climate change and the future of the planet, he switched to working on more practical topics and started the Azimuth Project, an international collaboration to create a focal point for scientists and engineers interested in saving the planet. His research now focuses on the math of networks and information theory, which should help us understand the complex systems that dominate biology and ecology.

Luke Muehlhauser: In a previous interview, I asked Scott Aaronson which “object-level research tactics” he finds helpful when trying to make progress in theoretical research, and I provided some examples. Do you have any comments on the research tactics that Scott and I listed? Which recommended tactics of your own would you add to the list?

John Baez: What do you mean by “object-level” research tactics? I’ve got dozens of tactics. Some of them are ways to solve problems. But equally important, or maybe more so, are tactics for coming up with problems to solve: problems that are interesting but still easy enough to solve. By “object-level”, do you mean the former?

Luke: Both! Conceiving of — and crisply posing — good research problems can often be even more important than solving previously-identified research problems.

John: Okay. Here are some of my tactics.
(1) Learn a lot. Try to understand how the whole universe works, from the philosophical, logical, mathematical and physical aspects to chemistry, biology, and the sciences based on those, to the historical sciences such as cosmology, paleontology, archaeology and history, to the social sciences such as psychology, sociology, anthropology, politics and economics, to the aspects that are captured best in literature, art and music.
It’s a never-ending quest, and obviously it pays to specialize and become more of an expert on a few things – but the more angles you can take on any subject, the more likely you are to stumble on good questions or good answers to existing questions. Also, when you get stuck on a problem, or get tired, it can be really re-energizing to learn new things.
(2) Keep synthesizing what you learn into terser, clearer formulations. The goal of learning is not to memorize vast amounts of data. You need to do serious data compression, and filter out the noise. Very often people will explain things to you in crappy ways, presenting special cases and not mentioning the general rules, stating general rules incorrectly, and so on.
This process goes on forever. When you first learn algebraic topology, for example, they teach you. homology theory. At the beginner’s level, this is presented as a rather complicated recipe for taking a topological space and getting a list of groups out of it. By looking at examples you get insight into what these groups do: the nth one counts the n-dimensional holes, in some sense. You learn how to use them to solve problems, and how to efficiently compute them.
But later—much later, in my case—you learn that algebraic topology of this sort not really about topological spaces, but something more abstract, called “homotopy types”. This is a discovery that happened rather slowly. It crystallized around the 1968, when a guy named Quillen wrote a book on “homotopical algebra”. It’s always fascinating when this happens: when people in some subject learn that its proper object of study is not what they had thought!
But even this was just the beginning: a lot has happened in math since the 1960s. Shortly thereafter, Grothendieck came along and gave us a new dream of what homotopy types might actually be. Very roughly, he realized that they should show up naturally if we think of “equality” as a process—the process of proving two thing are the same—rather than a static relationship.
I’m being pretty vague here, but I want to emphasize that this was a very fundamental discovery with widespread consequences, not a narrow technical thing.
For a long time people have struggled to make Grothendieck’s dream precise. I was involved in that myself for a while. But in the last 5 years or so, a guy named Voevodsky made a lot of progress by showing us how to redo the foundations of mathematics so that instead of treating equality as a mere relationship, it’s a kind of process. This new approach gives an alternative to set theory, where we use homotopy types right from the start as the basic objects of mathematics, instead of sets. It will take about a century for the effects of this discovery to percolate through all of math.
So, you see, by taking something important but rather technical, like algebraic topology, and refusing to be content with treating it as a bunch of recipes to be memorized, you can dig down into deep truths. But it takes great persistence. Even if you don’t discover these truths yourself, but merely learn them, you have to keep simplifying and unifying.
(3) Look for problems, not within disciplines, but in the gaps between existing disciplines. The division of knowledge into disciplines is somewhat arbitrary, and people put most of their energy into questions that lie squarely within disciplines, so it shouldn’t be surprising that many interesting things are lurking in the gaps, waiting to be discovered.
At this point, tactics (1) and (2) really come in handy. If you study lots of subjects and keep trying to distill their insights into terse, powerful formulations, you’re going to start noticing points of contact between these subjects. Sometimes these will be analogies that deserve to be made precise. Sometimes people in one subject know a trick that people in some other subject could profit from. Sometimes people in one subject have invented the hammer, and people in another have invented the nail—and neither know what these things are good for!
(4) Talk to lots of people. This is a great way to broaden your vistas and find connections between seemingly different subjects.
Talk to the smartest people who will condescend to talk to you. Don’t be afraid to ask them questions. But don’t bore them. Smart people tend to be easily bored. Try to let them talk about what’s interesting to them, instead of showing off and forcing them to listen to your brilliant ideas. But make sure to bring them some “gifts” so they’ll want to talk to you again. “Gifts” include clear explanations of things they don’t understand, and surprising facts—little nuggets of knowledge.
One of my strategies for this was to write This Week’s Finds, explaining lots of advanced math and physics. You could say that column is a big pile of gifts. I started out as a nobody, but after ten years or so, lots of smart people had found out about me. So now it’s pretty easy for me to blunder into any subject, write a blog post about it, and get experts to correct me or tell me more. I also get invited to give talks, where I meet lots of smart people.

Luke: You’ve explained some tactics for how to come up with problems to solve. Once you generate a good list, how do you choose among them?

John: Here are two bits of advice on that.
(1) Actually write down lists of problems.
When I was just getting started, I had a small stock of problems to think about – so small that I could remember most of them. Many were problems I’d heard from other people, but most of those were too hard. I would also generate my own problems, but they were often either too hard, too vague, or too trivial.
In more recent years I’ve been able to build up a huge supply of problems to think about. This means I need to actually list them. Often I generate these lists using the ‘data compression’ tactic I mentioned in part (2) of my last answer. When I learn stuff, I ask:

Is this apparently new concept or fact a special case of some concept or fact I already know?
Given two similar-sounding concepts or facts, can I find a third having both of these as special cases?
Can I use the analogy between X and Y to do something new in subject Y that’s analogous to something people have already done in subject X?
Given a rough ‘rule of thumb’, can I state it more precisely so that it holds always, or at least more often?

as well as plenty of more specific questions.
So, instead of being ‘idea-poor’, with very few problems to work on, I’m now ‘idea-rich’, and the challenge is keeping track of all the problems and finding the best ones.
I always carry around a notebook. I write down questions that seem interesting, especially when I’m bored. The mere act of writing them down either makes them less vague or reveals them to be hopelessly fuzzy. Sometimes I can solve a problem just by taking the time to state it precisely. And the act of writing down questions naturally triggers more questions.
Besides questions, I like ‘analogy charts’, consisting of two or more columns with analogous items lined up side by side. You can see one near the bottom of my 2nd article on quantropy. Quantropy is an idea born of the analogy between thermodynamics and quantum mechanics. This is a big famous analogy, which I’d known for decades, but writing down an analogy chart made me realize there was a hole in the analogy. In thermodynamics we have entropy, so what’s the analogous thing in quantum mechanics? It turns out there’s an answer: quantropy.
I later wrote a paper with Blake Pollard on quantropy, but I gave a link to the blog article because that’s another aspect of how I keep track of questions. I don’t just write lists for myself—I write blog articles about things that I want to understand better.
(2) Only work on problems when you think they’re important and you see how to solve them.
This tactic isn’t for everyone, but it works for me. When I was just getting started I would try to solve problems that I had no idea how to solve. People who are good at puzzles may succeed this way, but I generally did not.
It turns out that for me, a better approach is to make long lists of questions, and keep thinking about them on and off for many years. I slowly make progress until—poof!—I think I see something new and important. Only then do a take a problem off the back burner and start intensely working on it.
The physicist John Wheeler put it this way: you should never do a calculation until you already know the answer. That’s a bit of an exaggeration, because it’s also good to fool around and see where things go. But there’s a lot more truth to it than you might think.
Feynman had a different but related rule of thumb: he only worked on a problem when he felt he had an “inside track” on it—some insight or trick up his sleeve that nobody else had.

Luke: And once you’ve chosen a problem to solve, what are some of your preferred tactics for actually solving it?

John: By what I’ve said before, it’s clear that I get serious about a problem only after I have a darn good idea of how to solve it. At the very least, I believe I know what to do. So, I just do it.
But usually it doesn’t work quite that easily.
If you only officially tackle problems after absolutely every wrinkle has been ironed out by your previous musings, you’re being too cautious: you’ll miss working on a lot of interesting things. Many young researchers seem to fall prey to the opposite error, and waste time being completely stuck. The right balance lies in the middle. You break a problem down into sub-problems, and break those down into sub-subproblems… and you decide you’re ready to go when all these sub-subproblems seem likely to be doable, even before you’ve worked through the details.
How can you tell if they’re doable? This depends a lot on having previous experience with similar problems. If you’re a newbie, things that seem hard to you can be really easy to experts, while things that seem easy can turn out to be famously difficult.
Even with experience, some of sub-subproblems that seem likely to be routine will turn out to be harder than expected. That’s where the actual work comes in. And here it’s good to have lots of tricks. For example:
(1) If you can’t solve a problem, there should be a similar problem that’s a bit easier. Try solving that. And if you can’t solve that one… use the same principle again! Keep repeating until you get down to something you can solve. Then climb your way back up, one step at a time.
Don’t be embarrassed to simplify a problem to the point where you can actually do it.
(2) There are lots of ways to make a problem easier. Sometimes you should consider a special case. In math there are special cases of special cases of special cases… so there’s a lot of room for exploration here. If you see how enough special cases work, you’ll get ideas that may help you for your original problem.
(3) On the other hand, sometimes a problem becomes simpler when you generalize, leaving out potentially irrelevant details. Often people get stuck in clutter. But if it turns out the generalization doesn’t work, it may help you see which details were actually relevant.
(4) Sometimes instead of down or up the ladder of generality it pays to move across, by considering an analogous problem in a related field.
(5) Finally, a general hint: keep a written record of your efforts to solve a problem, including explanations of what didn’t work, and why. Look back at what you wrote from time to time. It’s amazing how often I come close to doing something right, forget about it, and come back later—sometimes years later—and see things from a slightly different angle, which makes everything fall into place. Failure can be just millimeters from success.

Luke: Thanks, John!
The post John Baez on Research Tactics appeared first on Machine Intelligence Research Institute.

John Baez on Research Tactics