Harry Buhrman on quantum algorithms and cryptography

 Harry Buhrman is head of the research group 'Algorithms and Complexity' at the Centrum Wiskunde & Informatica, Amsterdam, which he joined in 1994. Since 2000 he also has a joint appointment as full professor of computer science at the University of Amsterdam. Buhrman's research focuses on quantum computing, algorithms, complexity theory, and computational biology. In 2003 he obtained a prestigious Vici award and was coordinator of several national and international projects.
Buhrman is editor of several international journals and is member of various advisory and scientific boards, such as the advisory board of the Institute for Quantum Computing (Waterloo, Canada).

Luke Muehlhauser: In theory, position-based quantum cryptography could ensure that certain tasks could only be performed at certain locations:
For example, a country could send a message in such a way that it can only be deciphered at the location of its embassy in another country.
In Buhrman et al. (2011), you and your co-authors showed that
if collaborating adversaries are allowed to pre-share an arbitrarily large entangled quantum state, then… position-based cryptography… is impossible… in the quantum setting.
On the positive side, we show that for adversaries that are restricted to not share any entangled quantum states, secure position-verification is achievable. Jointly, these results suggest the interesting question whether secure position-verification is possible in the bounded case of a bounded amount of entanglement.
What is the current state of our knowledge on whether secure position-verification is possible given a bounded amount of entanglement?

Harry Buhrman: Classically such schemes will always be insecure, also if one restricts the running time or other resources of the adversaries. Quantumly the situation is more complicated, although we and others have shown that with an exponential amount of entanglement, also quantum schemes can be broken and are insecure. The quest is thus on for schemes that 1) can be easily implemented by the honest parties 2) require unreasonable amounts of entanglement for adversaries to break the scheme. We have seen some partial progress along these lines. For example a paper by Beigi and Koenig shows that there are schemes where a linear amount of entanglement is necessary to break them. It turns out that the question whether such schemes exists and if they do, prove for certain schemes that they are secure, is a very hard problem that we are currently working on. It turns out that the approach we have taken implies, if successful, results in classical complexity theory, and thus are very hard. On the other hand, it may still be possible that there simply do not exist schemes that are secure in this sense.

Luke: Based on your knowledge of the current state of quantum computing and quantum algorithm design, what recommendations to governments or citizens might you make?
E.g. Ronald de Wolf recommended we switch to post-quantum cryptography for sensitive data, especially since "even though there's no QC yet, the security services… are probably hoovering up RSA-encrypted communications that they store for the time being, waiting for the QC that will allow them to decrypt these messages later."

Harry: Yes I agree with Ronald completely. Switch as soon as possible to quantum proof security (although we don't know what exactly is quantum proof, see next paragraph). Indeed, all of our electronic encrypted communication, such as e-mail, telephone conversations, etc. that are encrypted using RSA-like techniques, can not yet be decrypted, but NSA and other organisations collect these encrypted data and wait till the day comes when they can decipher them. For example when a sufficiently large quantum computer is built such data can be decrypted very easily. So if we want to keep our data secure for a reasonable amount of time we should switch to other encryption techniques and standards. This applies more to governments, banks, big companies, than to normal users, but still I feel that a lot of people don't realise this. The common incorrect way of approaching this is as follows: This quantum computer, if built at all, is still 15 years ahead of us and when that time comes we switch then to more secure encryption. This reasoning is of course incorrect.
So what encryption should we use instead? Problematic is that this requires much more research. However a few encryption schemes have been proposed that appear to be quantum proof. I suggest that we swith to those in combination with good old RSA. So double encryption. This way the encryption security remains at least the same as what it is now, but most likely it is also quantum proof. One caveat of this approach is that encryption and decryption become (much) more inefficient since RSA like techniques are conveniently fast. This is the price one has to pay to be secure well into the future. So one has to make a decision when encrypting. Is it only necessary to be secure in the next year, e.g. when transmitting a credit card number it is mostly fine if it is only secure for a year or two, or do we need security well into the future? In the latter case I advise to switch as soon as possible.

Luke: Are there additional recommendations you can make to governments or citizens, based on your knowledge of the current state of quantum computing and quantum algorithm design?

Harry: Cryptographic implications are the ones that pertain immediately to society. What most people don't realise is that there are many problems in science and technology related fields, like eg chemistry, biology, physics and medicine design, that are computational very demanding and often impossible to carry out even on the most powerful supercomputers. Quantum computers wil help to solve some of these more efficiently notably when they concern simulation of quantum mechanical systems and properties. So research in QC will probably have an impact there as well.

Luke: From your perspective, what are the most significant factors blocking or slowing greater progress on the theoretical half (not the experimental half) of quantum computing research? In other words, what feasible but difficult changes to the field (or related fields) would most accelerate theoretical quantum computing research?

Harry: That is a difficult question to answer. If I knew what to change I would do it immediately. One of the problems is that we would like to have more quantum algorithms that demonstrate the power of quantum computation. The community has developed several techniques, like quantum Fourier sampling and quantum random walks. We would like to have more of those techniques and ideas. Maybe more important we also need to isolate computational problems or tasks that are amenable to quantum speedup. That is to say where quantum computers outperform classical computers.
Another area that could improve is that of fault tolerance and error correcting codes. A lot of progress has been made but we still don't know the exact value of how many hardware errors can be tolerated by algorithmic design. This value is called the fault tolerant  threshold. It  is important to know the exact value when  implementing qubits and running quantum  algorithms and protocols on them. We  will need to perform error correction and fault tolerant computation that fight the errors that popup due to imperfections and noise. But if the errors that occur are too big such correction by computation is not possible. Also it will be important to keep the overhead as small as possible.
A third problem is that of few qubit applications. We are beginning to see the demonstration of few qubit computers in the lab, but there are no good problems that could be used to demonstrate the usefulness of small quantum computers that operate on say 30-100 qubits.

Luke: In Buhrman et al. (1997), you and your co-authors listed 6 open problems in computational complexity theory. Have any of them been solved in the intervening time, perhaps at least with relativized results?

Harry: In that paper we listed 6 hypotheses that all are probably false. The goal was and still is to show that they are equivalent to P=NP. We also listed 6 open problems of which two were solved the next year by D. Sivakumar (# 2 and 5 from that list). As far as I know not much progress has been made towards these problems or showing that all the hypothesis are equivalent to P=NP. I think this is indication for how difficult the P versus NP question is. There currently is a new approach to tackling this problem, not the six hypotheses but the P vs NP problem itself. This approach is called geometric complexity and involves algebraic geometry and representation theory. Difficult topics in Mathematics.
There actually is a connection between quantum information and these types of classical questions. The language of quantum information allows one to state some of these problems in a different language which allows one to approach the problem from a different angle. In general this has been quite successful. For example the result of Ronald de Wolf and co-authors which uses quantum communication at its core, or the paper I have with Jop Briet, Troy Lee and Thomas Vidick which settles a problem in Banach space theory by solving a problem in quantum information. There also is a survey about other such results.
Perhaps we should revisit the 6 hypotheses paper with quantum techniques and see if we can make a little progress. Though at the moment I don't see a clear plan of attack.

Luke: Thanks, Harry!
The post Harry Buhrman on quantum algorithms and cryptography appeared first on Machine Intelligence Research Institute.


New paper: "Problems of self-reference in self-improving space-time embedded intelligence"

We've released a new working paper by Benja Fallenstein and Nate Soares, "Problems of self-reference in self-improving space-time embedded intelligence."
Abstract:
By considering agents to be a part of their environment, Orseau and Ring's space-time embedded intelligence is a better fit for the real world than the traditional agent framework. However, a self-modifying AGI that sees future versions of itself as an ordinary part of the environment may run into problems of self-reference. We show that in one particular model based on formal logic, naive approaches either lead to incorrect reasoning that allows an agent to put off an important task forever (the procrastination paradox), or fail to allow the agent to justify even obviously safe rewrites (the Löbian obstacle). We argue that these problems have relevance beyond our particular formalism, and discuss partial solutions.
This working paper also cites a brief new technical report by Fallenstein, "Procrastination in probabilistic logic."
Update 05/14/14: This paper has been accepted to AGI-14.
The post New paper: "Problems of self-reference in self-improving space-time embedded intelligence" appeared first on Machine Intelligence Research Institute.

New paper: “Problems of self-reference in self-improving space-time embedded intelligence”


Liveblogging the SV Gives Fundraiser

Today MIRI is participating in a massive 24-hour fundraiser called SV Gives. Strategy details here, donate here.
This blog post will be updated many times throughout the day.
Total donated to MIRI: $110,245.
Total prizes & matching won for MIRI: $61,330.

12:12am Pacific: That's a wrap, folks! Our thanks once again to SVCF for putting together a thrilling fundraiser with a neck-and-neck race toward the end. Our thanks to everyone who participated in SV Gives. And of course many thanks to our dedicated, well-organized donors who made this such a surprising success for MIRI. And finally, my thanks to the MIRI staff who worked very long hours to run the fundraiser, especially Malo Bourgon, who worked 38 hours straight, and now must go to sleep.
12:07am Pacific: For the last hour, MIRI once again won both the $2,000 prize and the $150 prize! Total raised is $110,245 plus $61,330 in prizes and matching funds!
12:04am Pacific: Fundraiser complete! $7.9M raised for local nonprofits. Congrats to SVCF for a rather thrilling fundraiser, and to leaderboard winners Sankara Eye Foundation and Burlingame Community for Education!
11:58pm Pacific: With two minutes to spare, we passed 2000 total donations today!
11:06pm Pacific: Another $2,000! And also another $150 for a random winner, Samuel Brooks!
10:10pm Pacific: And another $2,000!
9:22pm Pacific: MIRI has won $2,000 again!
8:07pm Pacific: MIRI wins $2,000 again!
7:57pm Pacific: Wow, MIRI and Sankara are still neck-and-neck for Total Unique Donors! Right now Sankara has a 5 person lead.
7:09pm Pacific: MIRI won another $2,000 for unique donors last hour, plus $6,980 in matching! But in other news, Sankara Eye Foundation has now overtaken us in the race for the Grand Prize!
6:21pm Pacific: MIRI won $3,030 in matching at 6pm. Thanks, everyone!
6:05pm Pacific: MIRI won another $2,000 for unique donors last hour. Great job everyone! Get ready for more matching funds being avilable at 7pm sharp!
5:54pm Pacific: Hurray! MIRI was one of 50 organizations to win $1,000 from the 5pm matching funds. The 6pm matching funds are coming up, so try to send your donation in the first ~2 seconds after the hour!
5:11pm Pacific: We won the hour again ($2,000), and we also won a 2nd randomly chosen $150 Golden Ticket, courtesy of a donation by Simon Safar!
4:36pm Pacific: Amazingly, MIRI is now 12% of all donations so far during SV Gives day.
4:27pm Pacific: We've reached 250 unique donors for the day; thanks so much! As it happens, there's a MIRI research workshop going on right now, so here is a photo of what you're all funding (several times over)!
4:06pm Pacific: Another $2,000 for MIRI. Note that we're coming up on the 5pm matching opportunity, so you should try to donate in the first 5 seconds after 5pm to try to win the match!
3:16pm Pacific: KFOX will be covering SV Gives between 5-7pm today.
3:06pm Pacific: Another $2,000! The strategy is still working!
2:10pm Pacific: MIRI won the hourly $2,000 again. Thanks, everyone!
1:36pm Pacific: Noon matching results have been released. MIRI got $2910 of the $50,000 in matching, which was apparently all taken in less than 4 seconds!
1:11pm Pacific: Another $2,000 prize won. Our deep and sincere thanks to SVCF for organizing SV Gives, and to MIRI's donors for being so dedicated and well-organized!
12:06pm Pacific: And we won the $2,000 again! Many thanks to our astoundingly well-coordinated donors.
11:07am Pacific: We've won the $2,000 Most Unique Donors Each Hour prize again. Who will break MIRI's winning streak?
10:34am Pacific: We've hit the 200 unique donors (for the whole day) mark!
10:08am Pacific: And yet another $2,000 prize for Most Unique Donors Each Hour! I bet the very gracious Sobrato Family Foundation wasn't expecting to be one of MIRI's largest donors of the year so far.
9:29am Pacific: On Twitter, SVCF asked us: "What's your secret? You've just got the 9 a.m. $2,000 Golden Ticket prize for a sixth time! Congrats!" We replied: "Our secret is: 70+ person-hours of planning and donor coordination, 10+ energy drinks, and many enthusiastic supporters!" Seriously, I think our participation in SV Gives should be sponsored by Rockstar energy drinks, given how many of them we are consuming over here!
9:07am Pacific: MIRI wins another $2,000. Three cheers for donor coordination!
8:22am Pacific: $1,970 more in matching came in at the beginning of this hour! (The first 14 seconds of this hour, to be precise.)
8:10am Pacific: Another hour, another $2,000 prize for MIRI! Thanks again, everyone.
7:48am Pacific: In the first ~30 seconds after 7am, we captured $2,340 in matching. Thanks, everyone!
7:07am Pacific: MIRI wins the $2,000 prize once again! (For the 6am-7am hour.)
6:16am Pacific: 150 unique donors today so far.
6:08am Pacific: Excellent work, everyone! MIRI won the $2,000 prize for "Most Unique Donors Each Hour" for a third time! (For the 5am-6am hour.)
5:27am Pacific: According to SV Gives' twitter account, NBC will be covering the SV Gives fundraiser in 3 minutes.
5:18am Pacific: Our thanks to Mick Porter, whose donation won the $150 prize for "random winner" during the 4am hour! We also won the $2,000 "Most Unique Donors Each Hour" again for the 4am-5am hour!
4:41am Pacific: So far we've won the $2,000 prize for Most Unique Donors Each Hour once, but not yet twice, so it looks like marginal donors giving as little as $10 each hour can make a difference!
3:50am Pacific: If you're visiting this page due to MIRI's SV Gives participation, and would like to know what it looks like to do technical work on ensuring good outcomes from future self-improving AI programs, check out our just-released new working paper, "Problems of self-reference in self-improving space-time embedded intelligence."
3:04am Pacific: Here's a live feed of MIRI HQ. Louie, Malo, myself, and Nate are there now.
2:23am Pacific: We won the $2,000 Golden Ticket for the 1am-2am hour! We also won the $2500 "Local Area Code Prize" by way of Patrick LaVictoire making the 408th donation (to any charity) of the day! Our thanks to the Sobrato Family Foundation and the Shackleton Family Foundation, respectively.
The post Liveblogging the SV Gives Fundraiser appeared first on Machine Intelligence Research Institute.

Liveblogging the SV Gives Fundraiser


Calling all MIRI supporters for unique giving opportunity!

Update: We are liveblogging the fundraiser here.
Read our strategy below, then give here!

As previously announced, MIRI is participating in a massive 24-hour fundraiser on May 6th, called SV Gives. This is a unique opportunity for all MIRI supporters to increase the impact of their donations. To be successful we'll need to pre-commit to a strategy and see it through. If you plan to give at least $10 to MIRI sometime this year, during this event would be the best time to do it!
The plan
We need all hands on deck to help us win the following prize as many times as possible:
$2,000 prize for the nonprofit that has the most individual donors in an hour, every hour for 24 hours.
To paraphrase, every hour, there is a $2,000 prize for the organization that has the most individual donors during that hour. That's a total of $48,000 in prizes, from sources that wouldn't normally give to MIRI.
The minimum donation is $10, and an individual donor can give as many times as they want. Therefore we ask our supporters to:

give $10 an hour, during every hour of the fundraiser that they are awake (I'll be up and donating for all 24 hours!);
for those whose giving budgets won't cover all those hours, see below for list of which hours you should privilege; and
publicize this effort as widely as possible.

International donors, we especially need your help!
MIRI has a strong community of international supporters, and this gives us a distinct advantage! While North America sleeps, you'll be awake, ready to target all of the overnight $2,000 hourly prizes.
Hours to target in order of importance
To increase our chances of winning these prizes we want to preferentially target the hours that will see the least donation traffic from donors of other participating organizations. Below are the top 12 hours we'd like to target in order of importance. Remember that all times are in Pacific Time. (Click on an hour to see what time it is in your timezone.)

1 am hour  (01:00–01:59 PT)
2 am hour (02:00–02:59 PT)
3 am hour (03:00–03:59 PT)
4 am hour (04:00–04:59 PT)
5 am hour (05:00–05:59 PT)
6 am hour (06:00–06:59 PT)
11 pm hour (23:00–23:59 PT)
7 am hour (07:00–07:59 PT)
10 pm hour (22:00–22:59 PT)
8 am hour (08:00–08:59 PT)
5 pm hour (17:00–17:59 PT)
9 pm hour (21:00–21:59 PT)

For the 5 pm hour there is an additional prize I think we can win:
$1,000 golden ticket added to the first 50 organizations receiving gifts in the 5 pm hour.
So if you are giving in the 5 pm hour try and give right at the beginning of the hour.
Bottom line, for every hour you are awake, give $10 an hour.
 Give preferentially to the hours above, if unable to give during all waking hours.
We also have plans to target the $300,000 in matching funds up for grabs during the event. If you would like to contribute $500 or more to this effort, shoot me an email at <EMAIL>.
For those who want to follow along and contribute to the last minute planning, as well as receive updates and giving reminders during the event, sign up below.

The post Calling all MIRI supporters for unique giving opportunity! appeared first on Machine Intelligence Research Institute.


Kasper Stoy on self-reconfigurable robots

 Kasper Stoy is a robotics and embodied artificial intelligence researcher holding an associate professor position at the Software and Systems Section of the IT University of Copenhagen. He is interested in the construction and design of complete robot systems, but being a computer scientist he has made most of his personal contributions in distributed control of multi-robot systems and modular robotics. He has published more than sixty papers in international conference proceedings or journals and is the author of the book "Self-Reconfigurable Robots: An Introduction" published by MIT Press. He also co-founded Universal Robots, a company focuses on user-friendly robot arms for industry. He is an active player in the international robot research community and reviews for all major journals and conferences in robotics. He has stayed for extended periods at University of Southern California, Harvard University, University of Tarapacá (Chile), and Seam Reap (Cambodia). He holds a M.Sc. degree in computer science and physics from the University of Aarhus, Denmark (1999) and a Ph.D. degree in computer system engineering from the University of Southern Denmark (2003) where he also worked as assistant professor (2003-2006) and associate professor (2006-2013). He is married and has two kids.

Luke Muehlhauser: In Larsen et al. (2013), you and your co-authors write:
Using a bottom-up, model-free approach when building robots [is] often seen as a less scientific way, compared to a top-down model-based approach, because the results are not easily generalizable to other systems… In this paper we will show how the use of well-known experimental methods from bio-mechanics are used to measure and locate weaknesses in our bottom-up, model-free implementation of a quadruped walker and come up with a better solution.
From looking at the paper I could see how your experimental method allowed you to find a better solution for your walker robot, but I couldn't understand how you addressed the challenge of generalizing the solution to other systems despite the bottom-up, model-free approach. Could you explain that part?

Kasper Stoy: The problem is that researchers who are doing cutting edge robotics want to explore how materials and their interaction can aid the movement of the robot. For instance, researchers have been working on passive-dynamic walkers for a long time now that exploit the mechanical system to achieve walking without using sensors or actuators. The energy comes from walking downhill and the control is open-looped – the mechanical system itself is self-stabilising. For these systems our current engineering approach is ok, but not great. We can just about model these kinds of walkers so we can get a good guess of the initial parameters to get the system walking, but there is still an extended phase of tinkering before the system actually walks. It is in this tinkering phase that our precise motion capture as used in biomechanics comes into play. Given measured paths of all parts of the robot we can analyse the data and come up with reasonable hypotheses about how to improve the robot. Hence, the tinkering becomes much more systematic even though the underlying physical processes are too complex to be modelled. It may not be apparent, but just modelling the impact of a foot with the ground which takes all types of frictions into account, the deformation of the foot, the spring effect, and so on is largely intractable. Hence, the underlying assumption of our work is that all models of locomotion are fundamentally wrong. They may give a high-level picture of what is going on, but fundamentally they cannot be used to predict what will happen just two steps later. However, if we turn this around and we have gotten our robot to walk and we record the data of how it walks. Although difficult, we can build a model that match this data which is based on the ground truth and where there is no modelling bias on the part of the researcher. We now have a better informed model that can be used to build the next generation of robots. Hence, the model is a generalisation of our specific implementation which you can copy, but you can also replace elements of which you have better implementations. In locomotion research like many other fields the models and physical systems are drifting apart because it is researchers with different skill and interests who work on them. In our work, we are clearly working on physical systems and just provide a hint to how our experimental results can be generalised in a way that is meaningful to modelling oriented researchers. We hope.

Luke: You co-authored a book on self-reconfigurable robots in 2010. What do you mean by that term, and what are some recent examples of practical self-reconfigurable robots? (Perhaps, more recent than the book?)

Kasper: Self-reconfigurable robots are an idea inspired by multi-cellular organisms. The idea is that instead of building a robot as a single expensive, monolithic, and fragile piece of hardware you would built a robot from many relative simple robotic cells. We refer to these robotic cells as modules and the research community as modular robotics. Self-reconfigurable robots then take this a step further in that not only is the robot multi-cellular, the modules can also rearrange the way they are connected automatically. Hence as the modules wander around each other the robot as a whole change shape. While this is rare in biology there is a small aquatic animal called hydra which has this ability. In fact, it has been shown that if you cut it in half it transform itself into two smaller replicas of the original hydra. The advantage of this type of robot is extreme robustness and versatility. Features that are particular important in for instance extra-terrestrial exploration. Regarding, application this was from the beginning curiosity driving research. Is it possible to build a robot that is multi-cellular and able to chance its own shape? The answer is yes. The concept has been implemented successfully in about five systems internationally. However, modules are relatively expensive, big (10cm diameter), complex, and quite heavy. Hence, direct applications are still lacking given that the cost-benefit of these robots is not good enough for a commercial market. However, the more modest modular robots have seen some recent success on KickStarter with the Modbots. A modular construction kit for building your own robots.

Luke: What are each of the "about five systems" you refer to?

Kasper: The most developed system is the M-TRAN III self-reconfigurable robot developed at AIST in Japan. The runner ups are ATRON which was developed mostly by my former colleagues at University of Southern Denmark. However, both of the above systems are not being developed further. Another runner up is the Roombots developed at EPFL in Switzerland. Recent exciting development are the M-Blocks coming of of MIT and SMORES coming ot of UPenn. There is a large number of robots which come very close, but not quite achieving convincing three dimensional self-reconfiguration.
Also, check out this Wikipedia article for a more comprehensive list.

Luke: What are some specific breakthroughs or kinds of progress that you expect to see in the next 10-15 years of research into self-reconfiguring modular robots?

Kasper: The field has very much been about whether it is possible to build robots that can change their own shape or not. Within the last decade, we have established that this is indeed the case. However, currently the robotic cells are too complex, heavy, and expensive for most applications. Hence, one of the goals is to reduce all of these obstacles, but given the current state of the art in mechatronics this is quite difficult. Hence, I see a mainstream area of research focussing on various simplifications of the mechatronics to make it attractive for specific applications in space, robotic prototyping tools and the like. However, it also clear that other implementations have to be considered. For this some have turned to DNA-based implementations or using micro fabrication technology. Another unexplored opportunity is in soft robotics that is gain traction in the robotic community as a whole. The question is then where this would lead in 10-15 years. I think all areas have potential so if for instance the soft modular robotics effort is successful this could lead to robotic multi-cellular systems much closer to biological cells in terms of material properties. Where would this find applications is anyones guess, but maybe in furniture, clothing or in healthcare in the form of adaptive casts.

Luke: Thanks, Kasper!
The post Kasper Stoy on self-reconfigurable robots appeared first on Machine Intelligence Research Institute.


MIRI's May 2014 Newsletter

On May 6th, there is $250,000+ in matching funds and prizes available from sources that normally wouldn't contribute to MIRI at all. Details here.

Research Updates

New paper: "The errors, insights, and lessons of famous AI predictions."
Botworld update with 4 new games, including Prisoner's Dilemma and Stag Hunt.
12 new expert interviews, including e.g. Suzana Herculano-Houzel on cognitive ability and brain size.

News Updates

Can We Really Upload Johnny Depp's Brain? (in Slate)
Why MIRI?

Other Updates

Call for Papers: Special Issue. Confronting Future Catastrophic Threats To Humanity
AI Risk, New Executive Summary by Stuart Armstrong
Transcending complacency on superintelligent machines, by Hawking, Tegmark, Russell, and Wilczek

As always, please don't hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI's May 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s May 2014 Newsletter


New Paper: "The errors, insights, and lessons of famous AI predictions"

During his time as a MIRI researcher, Kaj Sotala contributed to a paper now published in the Journal of Experimental & Theoretical Artificial Intelligence: "The errors, insights and lessons of famous AI predictions – and what they mean for the future."
Abstract:
Predicting the development of artificial intelligence (AI) is a difficult project – but a vital one, according to some analysts. AI predictions are already abound: but are they reliable? This paper starts by proposing a decomposition schema for classifying them. Then it constructs a variety of theoretical tools for analysing, judging and improving them. These tools are demonstrated by careful analysis of five famous AI predictions: the initial Dartmouth conference, Dreyfus's criticism of AI, Searle's Chinese room paper, Kurzweil's predictions in the Age of Spiritual Machines, and Omohundro's 'AI drives' paper. These case studies illustrate several important principles, such as the general overconfidence of experts, the superiority of models over expert judgement and the need for greater uncertainty in all types of predictions. The general reliability of expert judgement in AI timeline predictions is shown to be poor, a result that fits in with previous studies of expert competence.
The post New Paper: "The errors, insights, and lessons of famous AI predictions" appeared first on Machine Intelligence Research Institute.

New Paper: “The errors, insights, and lessons of famous AI predictions”


Suresh Jagannathan on higher-order program verification

 Dr. Suresh Jagannathan joined DARPA in September 2013. His research interests include programming languages, compilers, program verification, and concurrent and distributed systems.
Prior to joining DARPA, Dr. Jagannathan was a professor of computer science at Purdue University. He has also served as visiting faculty at Cambridge University, where he spent a sabbatical year in 2010; and as a senior research scientist at the NEC Research Institute in Princeton, N.J.
Dr. Jagannathan has published more than 125 peer-reviewed conference and journal publications and has co-authored one textbook. He holds three patents. He serves on numerous program and steering committees, and is on the editorial boards of several journals.
Dr. Jagannathan holds Doctor of Philosophy and Master of Science degrees in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology. He earned a Bachelor of Science degree in Computer Science from the State University of New York, Stony Brook.

Luke Muehlhauser: From your perspective, what are some of the most interesting or important developments in higher-order verification in the past decade?

Suresh Jagannathan: I would classify the developments of the past decade into four broad categories:

The development of higher-order recursion schemes (or higher-order model checking). This approach is based on defining a language semantics in terms of a (recursive) tree grammar. We can now ask whether the tree generated by the grammar satisfies a particular safety property. Because recursion schemes are very general and expressive structures, they can be viewed as natural extensions of model checkers based on pushdown automata or finite-state systems. In the past decade, there have been substantial advances that have made higher-order model checking a practical endeavor, accompanied by realistic implementations.
Liquid Types combine facets of classic type inference techniques found in functional language with predicate abstraction techniques used in model checkers to automatically infer dependent type refinements with sufficient precision to prove useful safety properties about higher-order functional programs. Implementations of this approach found in OCaml, SML, and Haskell have demonstrated that it is indeed possible to verify non-trivial program properties without significant type annotation burden.
There have been substantial advances in the development of new languages built around rich dependent type systems that enable the expression and static verification of rich safety and security specifications and properties of higher-order programs. These include the languages that support mechanized proof assistants like Coq, Agda, or Epigram, languages like F* geared towards secure distributed programming, libraries like Ynot that encapsulate imperative (stateful) features and Hoare-logic pre/post-conditions within a higher-order dependent type framework, and features such as GADTs and type classes found in languages like GHC.
Polyvariant control-flow analyses like CFA2 or higher-order contracts are two techniques that facilitate verification and analysis of dynamic higher-order languages. CFA2 adopts pushdown models used in program analyses for first-order languages to a higher-order setting, enabling precise matching of call/return sequences. Higher-order contracts allow runtime verification and blame assignment of higher-order programs and are fully incorporated into Racket, a multi-paradigm dialect of Lisp/Scheme.

Luke: What are some of the "realistic implementations" of higher-order model checking that you refer to?

Suresh: The most scalable implementation to date is the tool Preface (Ramsay et. al, POPL'14) that is capable of handling recursion schemes of several thousand rules (i.e., function definitions). Earlier work such as TRecS (Kobayshi, POPL'09) have good performance for recursion schemes of up to hundreds of rules.

Luke: What are your recommended readings on recent interesting work on liquid types and on polyvariant control-flow analyses?

Suresh:
Liquid Types

Vazou, Rondon, Jhala, "Abstract Refinement Types", European Symposium on Programming, 2013, p. 209-228
Rondon, Kawaguchi, Jhala, "Low-Level Liquid Types", ACM Symposium on Principles of Programming Languages (POPL), 2010, pp. 131-144.
Rondon, Kawaguchi, Jhala, "Liquid Types", ACM Conference on Programming Language Design and Implementation, 2008, pp. 159-169.
Kawaguchi, Rondon, Bakst, Jhala, "Deterministic Parallelism via Liquid Effects", ACM Conference on Programming Language Design and Implementation, 2012, pp. 45-54.
Zhu, Jagannathan, "Compositional and Lightweight Dependent Type Inference for ML", Conference on Verification, Model-Checking and Abstract Interpretation, 2013, pp. 295-314.

Polyvariant Control-Flow Analysis

Midtgaard, "Control-Flow Analysis of Functional Programs", ACM Computing Surveys, 2012, 44(3).
Earl, Sergey, Might, Van Horn, , "Introspective Pushdown Analysis of Higher-Order Programs", International Conference on Functional Programming, 2012, pp. 177-188.
Might, Smaragdakis, Van Horn, "Resolving and Exploiting the k-CFA Paradox: Illuminating Functional vs. Object-Oriented Program Analysis", ACM Conference on Programming Langauge Design and Implementation, 2010, pp. 305-315.
Vardoulakis, Shivers, "CFA2: A Context-Free Approach to Control-Flow Analysis", Logical Methods in Computer Science, 2011, 7(2).

Luke: What are some key developments and solutions you hope to see in the next 5-10 years of research into higher-order verification techniques? Which breakthroughs could be most important, if they could be achieved, given our current understanding?

Suresh: Currently, much of the work on higher-order verification centers around foundations (the definition of expressive models for describing salient properties that arise in HO programs, as is the case with current work in higher-order recursion schemes, dependent type systems, or control-flow analysis), or pragmatics (reducing type annotation burden as in Liquid Types). Going forward, I would expect to see a convergence of these techniques that enable applying new expressive and foundational models at scale.
One important breakthrough would be the integration of these systems into realistic compilers that would go beyond simple safety and property checking, to inform new kinds of verifiable optimizations and program transformations. One can envision using these techniques to use the specifications HO verification techniques support, as a way of realizing tangible code improvements and performance gains in verified compilers and runtime systems. Another important advance would be a deeper understanding of the potential synergistic interplay between higher-order verification techniques and other kinds of specification systems (e.g., session types). The issues here deal with how we would express rich protocols that capture not only safety requirements, but liveness and other kinds of sophisticated modalities.

Luke: Thanks, Suresh!
The post Suresh Jagannathan on higher-order program verification appeared first on Machine Intelligence Research Institute.


Ruediger Schack on quantum Bayesianism

 Ruediger Schack is a Professor at the Department of Mathematics at Royal Holloway, University of London. He obtained his PhD in Theoretical Physics at the University of Munich in 1991 and held postdoctoral positions at the Max Planck Institute for Quantum Optics, the University of Southern California, the University of New Mexico, and Queen Mary and Westfield College before joining Royal Holloway in 1995. His research interests are quantum information theory, quantum cryptography and quantum Bayesianism.

Luke Muehlhauser: In Fuchs et al. (2013), you and your co-authors provide an introduction to quantum Bayesianism aka "QBism," which you more or less co-invented with Carlton Caves and Christopher Fuchs. But before I ask about QBism, let me ask one of the questions asked of the interviewees in Elegance and Enigma: The Quantum Interviews (including Fuchs): "What first stimulated your interest in the foundations of quantum mechanics?"

Ruediger Schack: I can trace the beginning of my interest in quantum foundations to reading one paper: "Where do we stand on maximum entropy?" by Ed Jaynes, and one book: Du Microscopique au Macroscopique by Roger Balian. Jaynes's paper introduced me to Bayesian probability theory, and Balian's book taught me that one can think of quantum states as representing Bayesian probabilities.

Luke: What is your own summary of the message of QBism? And what do you think its practical import for the world could be?

Ruediger: In two words the message of QBism is that people matter. According to QBism, quantum mechanics is a theory that any agent can use to organize his experience. More precisely, quantum mechanics permits any agent to quantify, on the basis of his past experiences, his probabilistic expectations for his future experiences. QBism takes measurement outcomes as well as quantum states to be personal to the agent using the theory. Quantum mechanics does therefore not provide an objective, agent-independent description of the world – it rules out a "view from nowhere". By clarifying thus the role of quantum mechanics and of science in general, QBism avoids all the interpretational difficulties usually associated with quantum foundations. In QBism, there are no objective elements of reality that determine either measurement outcomes or probabilities of measurement outcomes. Rather, every quantum measurement is an action on the world by an agent that results in the creation of something entirely new. QBism holds this to be true not only for laboratory measurements on microscopic systems, but for any action an agent takes on the world to elicit a new experience. It is in this sense that agents – people – have a fundamental creative role in the world.
Any interpretation of quantum mechanics by definition makes the same predictions as quantum mechanics. Nevertheless, I expect QBism to have practical import for the world. By shifting the focus away from interpretations that regard quantum states as real, QBism opens up new possibilities: in the search for a compelling physical principle that would explain the quantum formalism, and in the search for new physics.

Luke: I heard that Fuchs, at least, thinks that the case for QBism would be more compelling if it turns out that SIC-POVMs (symmetric informationally complete positive operator-valued measures) existed in every finite Hilbert space dimension, which is currently an unsolved question. Is that your understanding as well? If so, what's the reasoning?

Ruediger: QBism as an interpretation of quantum mechanics is independent of the existence of SICs and can be formulated without referring to SICs. But QBism is also a program, ultimately with the aim of discovering new physics. A more immediate goal is to find a simple and compelling physical principle underpinning the quantum formalism. Now one of QBism's central tenets is that a measurement does not reveal a preexisting outcome but results in the creation of something new. In the quantum formalism, this idea finds a simple expression in the fact that the classical probability sum rule does not apply to the – necessarily hypothetical – outcomes of an unperformed experiment. For instance, in the double-slit experiment the probability distribution for the measured particle position on the screen cannot be obtained by adding the weighted probabilities given the particle goes through one or the other slit.
So far this is a purely negative statement. If a SIC exists in every finite Hilbert-space dimension, it turns into a powerful positive statement. If the hypothetical measurement is a SIC measurement, the Born rule takes the form of a very simple modification of the probability sum rule. What is more, from the modified probability sum rule, a large part of the structure of quantum mechanics can be derived. In this picture, instead of the purely negative statement that the probability sum rule cannot be used, we would have a simple physically motivated principle that implies a substantial part of the quantum formalism. In that sense, the existence of SICs in all dimensions would strengthen the case for QBism.

Luke: Roughly how many people are actively advocating or contributing to QBism? Do you think it's particularly difficult to draw funding and cognitive talent toward this work because of its theoretical nature, or for other reasons?

Ruediger: As a matter of fact, the mathematical aspects of QBism (such as the structure of the SICs or the quantum de Finetti theorems) have attracted significant funding over the years. At present, however, only a small number of people are actively contributing to QBism. When QBism holds that science is as much about the scientist as it is about the world external to the scientist, it challenges one of the most deeply held prejudices that most physicists subscribe to. This prejudice is exemplified by the following quote from Landau and Lifshitz: "By measurement, in quantum mechanics, we understand any process of interaction between classical and quantum objects occurring apart from and independently of any observer." Another commonly held prejudice is that a probability-1 assignment implies the existence of an objective mechanism that brings about the event. Physicists find it very hard to accept the QBist principle that probability-1 judgments are still judgments, like any other probability assignments. Let me finish with a prediction: In twenty-five years when a new generation of scientists have been exposed to QBist ideas, QBism will be taken for granted and quantum foundations will have disappeared as a problem.

Luke: Thanks, Ruediger!
The post Ruediger Schack on quantum Bayesianism appeared first on Machine Intelligence Research Institute.

Ruediger Schack on quantum Bayesianism


David J. Atkinson on autonomous systems

 David J. Atkinson, Ph.D, is a Senior Research Scientist at the Florida Institute for Human and Machine Cognition (IHMC). His current area of research envisions future applications of intelligent, autonomous agents, perhaps embodied as robots, who work alongside humans as partners in teamwork or provide services. Dr. Atkinson's major focus is on fostering appropriate reliance and interdependency between humans and agents, and the role of social interaction in building a foundation for mutual trust between humans and intelligent, autonomous agents. He is also interested in cognitive robotics, meta-reasoning, self-awareness, and affective computing. Previously, he held several positions at California Institute of Technology, JPL (a NASA Center), where his work spanned basic research in artificial intelligence, autonomous systems and robotics with applications to robotic spacecraft, control center automation, and science data analysis. Recently, Dr. Atkinson delivered an invited plenary lecture on the topic of "Trust Between Humans and Intelligent Autonomous Agents" at the 2013 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2013). Dr. Atkinson holds a Bachelor's degree in Psychology from University of Michigan, dual Master of Science and Master of Philosophy degrees in Computer Science (Artificial Intelligence) from Yale University, and the Doctor of Technology degree (d.Tekn) in Computer Systems Engineering from Chalmers University of Technology in Sweden.

Luke Muehlhauser: One of your projects at IHMC is "The Role of Benevolence in Trust of Autonomous Systems":
The exponential combinatorial complexity of the near-infinite number of states possible in autonomous systems voids the applicability of traditional verification and validation techniques for complex systems. New and robust methods for assessing the trustworthiness of autonomous systems are urgently required if we are to have justifiable confidence in such applications both pre-deployment and during operations…  The major goal of the proposed research is to operationalize the concept of benevolence as it applies to the trustworthiness of an autonomous system…
Some common approaches for ensuring desirable behavior from AI systems include testing, formal methods, hybrid control, and simplex architectures. Where does your investigation of "benevolence" in autonomous systems fit into this landscape of models and methods?

David J. Atkinson: First let me point out that test, formal methods and such other techniques that you point out have little to do with ensuring desirable behavior and more to do with avoiding errors in behavior due to design or implementation flaws. These are not equivalent. Existing techniques improve reliability, but that is only one component of trust and it only concerned with behavior "as designed". Furthermore, as your quote from my material points out, when it comes to the near-infinite state spaces of autonomy, those "common approaches" are inherently limited and cannot make the same strong claims regarding system behavior that they could with machines where the envelope of behavior could be completely known.
Therefore, I chose to look for answers regarding trust, trustability and trustworthiness in the operations phase of the autonomous system lifecycle because it is here, not in testing and evaluation, where the full complexity of autonomous behavior will manifest itself in response to the uncertainty, dynamics, and real-world complexity that is its major strength. My approach is to focus is on the applicability of human interpersonal trust to operation of autonomous systems. The principle reasons for this are 1) Autonomy is limited — humans make the decisions to rely upon an intelligent agent, subject to a variety of individual and situational factors and constraints, and; 2) Eons of evolution have created a reasonably good mechanism in humans for trust. It is fundamental to every human social transaction and it works. Beyond reliability, studies in multiple disciplines have shown that people want evidence of capability, predictability, openness (aka transparency), and safety before granting a measure of trustworthiness to a machine. Key questions revolve around the nature of that evidence, how it is provided, and what inferences can be reasonably made. People use both cognitive and affective mental processes for evaluating trustworthiness.  My central claim is that if we can reverse engineer these mechanisms and make intelligent autonomous agents (IATs) that are compliant with the human trust process (two big ifs), then we will have created a new way for humans to have trusting relationships with they machines they rely upon. It will be a transition from seeing IATs as tools to treating them as partners.
Benevolence is interesting for a couple of reasons. First, it is a complex attribution built upon a structure of beliefs about another person that include good will, competency, predictability, lack of a hidden agenda, agency and other beliefs, most which are likely to play a role in many kinds of interactions between human and IAT. Second, just looking at that list of beliefs makes it a hard problem, although my colleagues think people will have no problem attributing agency to a machine. Third, there are important applications of IATs, perhaps embodied as robots, where the unique psychology of benevolence is critical to success. For example, disaster rescue. It is long known that disaster victims have a unique psychology borne of stress, fear and the psycho-physiological effects these evoke. Human first responders undergo special training on victim psychology. One of the reasons is that a victim, justifiably afraid for their life, may be very reluctant to trust a rescuer and without trust there is no cooperation and rescue can become very difficult. Benevolence seems to be a part of the trust that is required. Today, we have no idea whatsoever whether a real disaster victim will trust and cooperate with a robot rescuer.
Circling back to your question about "fitting in to the landscape of models and methods", the ultimate goal of my research is to formulate design requirements, interaction methods, operations concepts, guidelines and more that, if followed, will result in an IAT that can itself engender well-justified human trust.

Luke: As you say, formal methods and other techniques are of limited help in cases where we don't know how to formulate comprehensive and desirable design requirements, as is often the case for autonomous systems operating in unknown, dynamic environments. What kinds of design requirements might your approach to trustworthy systems suggest? Would these be formal design requirements, or informal ones?

David: There will certainly be gaps in what we can do today regarding the specific requirements related to unknown, dynamic environments. One of our objectives is to narrow those gaps so precise questions can be studied. It is one thing to wave hands and moan about uncertainty, and another entirely to do something about it.
Generally speaking, we are working towards formal specification of the traditional types of requirements: Functional, Performance, Design Constraint, and Interface (both Internal and External). By "formal", I mean "complete according to best practices". I do not mean "expressed in formal logic or according to a model-based language" — that is a step beyond. Requirements may be linked in numerous ways to each other forming a directed graph (hopefully with no cycles!). Relationships include Source, Required-By, Depends-upon and so forth.
For example, an attribution of benevolence by a human "Trustor" requires a belief (among numerous others) that the candidate "Trustee" (the autonomous system) has "no hidden ill will". This is a very anthropomorphic concept that some might scoff at, but studies have demonstrated its importance to attribution of benevolence and so it is a factor we must reckon in designing trustworthy and trustable autonomous systems. But what does it even mean?
Just to give you an idea of how we are breaking this down, here are some of the derived requirements. I should emphasize that it is very premature to make any claims about the quality or completeness of what we done with requirements engineering thus far — mostly that work is on the schedule for next year so I'll only give you the types and titles: We will have as a matter of course a generic Level 1 Functional requirement to "Provide Information to Human User". Derived from this is an Interface requirement something like "Volunteer Information: The Autonomous System shall initiate communication and provide information that is import to humans" (actually, that is two requirements). This in turn is elaborated by a number of Design Constraints such as "Disposition: The Autonomous System shall disclose any disposition that could result in harm to the interests of a human". These Design Constraints are in turn linked to numerous other detailed requirements such as this Performance Requirement "Behavior – Protective: The Autonomous System shall recognize when its behavior could result in harm to the interests of a human". A designer of an autonomous system will recognize a number of very hard problems in this simple example that need to be solved to effectively address the hypothetical need for a benevolent autonomous system.. Our focus in this project is on "what" needs to be done, not "how" to do it.
Without a doubt, this requirements process will generate plenty of questions that will require further study. It is also likely that some requirements may conflict and any particular application of autonomy must do tradeoff studies to prioritize. Nevertheless, our goal is to spell out as much as we can. We will differentiate mandatory requirements from goals or objectives. Where it is possible, we will allocate requirements to particular elements of an autonomous system's architecture, such as "goal selection mechanism" (where issues relating to prioritization may arise and affect predictability and therefore trust). For every requirement, we will provide a rationale with links to studies and empirical data or other discussion that can be used in analysis. That part is very high priority in my mind. Too many times I have encountered requirements where it is impossible to understand how they were derived. I'd also like to take a risk-driven view of the requirements so that individual requirements, or groups of requirements, can be associated with particular risks. That is another area that will have to be quantified by further application-specific analysis. Finally, a good requirement has to be verifiable. There is considerable work to be done on this topic with respect to autonomous systems.
Requirements engineering is a lot of work of course, and the history of large scale system development is replete with horror stories of poor requirements. That's why I want to express our trust-related requirements formally following requirements engineering standards  the extent possible. From my previous experience at NASA, I know that the less work a project has to do, the more likely it is to to adopt existing requirements. So we are developing our trust-related requirements consciously with the goal of making them easy to understand and easy to reuse. Finally, given the scope of what is required it is likely that we will only be able to go just so far under the auspices of our current project to provide a vector for future work (hint hint to potential sponsors out there!)

Luke: From your description, this research project seems somewhat interdisciplinary, and the methodology seems less clear-cut than is the case with many other lines of research aimed at similar goals (e.g. some new project to model-check a particular software design for use in robotics). It's almost "pre-paradigmatic," in the Kuhnian sense. Do you agree? If not, are there are other research groups who are using this methodology to explore related problems?

David: Yes, the project is very interdisciplinary but with the primary ones being social and cognitive psychology, social robotics, and artificial intelligence we well as the rigor contributed by solid systems engineering. The relevant content within each of those disciplines can be quite broad. It is not a small undertaking. My hope is that I can plant some memes in each community to help bring them together on this topic of trust and these memes will foster further work. As far as methodology, we are pursuing both theoretical development and experimentation, and trying to be as rigorous as exploratory work of this nature will permit. We have to understand the previous psychological work on human interpersonal trust, and human factors studies on human-automation trust, to find those results that may have important implications for human trust of an intelligent, autonomous agent. The importance of agency is a good example.
We know from psychological studies that an attribution of "free-will", or more narrowly, the ability to choose, is an important component of deciding whether someone else is benevolent or not. That is, if a person feels the other is compelled to help then the less likely they are to believe that other person is benevolent. Apart from philosophers, most people don't think very deeply about this. They make a presumption of free-will and then look to see if there are reasons it is limited, for example, is the person just following orders, or required by their profession or social norms to act in a particular way? With machines, we start from the other side: people assume machines have no free-will because they believe machines are (just) programmed. However, there are some studies that suggest that when machine behavior is complex (enough), and somewhat unpredictable because there are numerous possible courses of action, then people begin to attribute mental states including the ability to choose. My hunch is this is an evoked response of our innate demand to interpret the actions of others in a framework of intentionality. At some point, enough features are present that those evolution-designed heuristics kick in to aid in understanding. WHY that may be, I will leave to the evolutionary socio-biologists.
Back to methodology now: This is a phenomena for which we can design a study involving humans and machines, with systematic variation of various factors to see what qualities a machine actually requires in order to evoke a human attribution of the ability to choose to help. This year we will be conducting just such a study. I hope to begin running participants this summer. We have been rigorous with experimental design, choice of what data to collect and the statistical methods we will use to analyze the results. While we will fudge a little bit on the robot implementation, using simulation in places and "wizard-of-oz" techniques for language interaction for example, the products of the study ought to be recognized as solid science by researchers in multiple disciplines if we do it right. In general, I think this a very hard goal to achieve because each discipline community has certain preferences and biases about what they like to see in methodology before they are convinced. There are a couple of other groups working on social robotics who use this approach, and a very few number of psychologists who are working on human-centered design of automation. I don't want to start listing names because I'm sure there are others of whom I'm not yet aware. I do know that I am certainly not the first to confront this challenge. Multidisciplinary research of this type is always, in some sense, pre-paradigmatic because it is a struggle for understanding and legitimacy at the boundaries of separate disciplines, not the core. Artificial intelligence has always been multidisciplinary, a strength as well as a weakness as far as more general acceptance among related disciplines. I don't worry too much about what other people think. I just do what I believe has to be done.

Luke: What are some concrete outcomes you hope will be achieved during the next 5 years of this kind of research?

David: As a wise man once said, "It's hard to make predictions, especially about the future."
My hopes. This one is concrete to me, but perhaps not what you had in mind: I hope that the value of our approach will be convincingly demonstrated and other research groups and early-career researchers will join in. There is much to be explored and plenty of opportunity to make discoveries that can have a real impact.
On balance, it seems that in many potential application domains the biggest challenge is not mistrust or absence of trust, but excessive trust. It is abundantly clear that people have no trouble trusting machines in most contexts, and, as this is somewhat tied to generational attitudes, the condition will probably increase. Sometimes this leads to over-reliance and complacency, and then surprising (and potentially dangerous) conditions if things go sour. A funny but real example is the driver who set cruise control on his RV on a long stretch of straight highway, got out of the driver's seat, and went in back to make a sandwich. You can guess what happened. Effective teamwork and cooperation requires that each team member understands the strengths and limitations of the others so under- and over-reliance do not occur. Intelligent, autonomous teammates need that same ability to participate effectively in this essential team-building process — a process that takes time and experience to build mutual familiarity. We will contribute to that solution.
I believe our work will lead directly to an understanding of When, Why, What, and (some of) How a machine needs to interact with human teammates to inoculate against (and/or correct for) under- and over-reliance. This technology is key for solving what many people claim is a lack of transparency in intelligent systems. It will help "users" to better understand the competency of a machine (the most important quality) in a given context consisting of dynamic situational factors, tasks and goals. This will in turn increase predictability (another important quality) and thereby help mitigate concerns about risks and safety. Ultimately, a deep solution that is broadly applicable will require a higher degree of machine meta-reasoning and self-awareness than we can engineer today, but this is an area of active research where useful results ought to be appearing more and more frequently. (The field of cognitive (developmental) robotics is very exciting.) However, I do expect concrete and useful results for early applications in some semi-structured task domains. A few examples of domains containing "low hanging fruit" for applications are transportation (e.g., autonomy-assisted driving, long haul trucking), healthcare (patient monitoring, therapy assistance, assisted living), and some defense-related applications. My group is actively working towards all of these possibilities. I don't want to leave you with the impression that creating effective applications will be easy because many hard basic research challenges remain, and we will undoubtedly discover others when we start to transition the technology into real-world applications. Nevertheless, I'm optimistic!

Luke: Thanks, David!
The post David J. Atkinson on autonomous systems appeared first on Machine Intelligence Research Institute.

David J. Atkinson on autonomous systems


Help MIRI in a Massive 24-Hour Fundraiser on May 6th

Update: We're now liveblogging the fundraiser here.

On May 6th, MIRI is participating in Silicon Valley Gives. We were selected to participate along with other local Bay Area charities by the Silicon Valley Community Foundation. On this day, we recommend donors make gifts to MIRI through the SV Gives portal so we can qualify for some of the matching and bonus funds provided by dozens of Bay Area philanthropists.
Why is this exciting for supporters of MIRI? Many reasons, but here are a few:

Over $250,000 of matching funds and prizes up for grabs, from sources that normally wouldn't contribute to MIRI:

Kick-off Match! Two-to-one dollar match up to $50,000 during the midnight hour.
$2,000 prize for the nonprofit that has the most individual donors in an hour, every hour for 24 hours.
Golden Ticket of $150 added to a random donation each hour, every hour, for 24 hours.
Dollar for Dollar match up to $35,000 during the 7 AM hour.
Dollar for Dollar match up to $15,000 during the 8 AM hour.
Dollar for Dollar match up to $50,000 during the 12 noon hour.
Dollar for Dollar match up to $50,000 during the 6 PM hour.
Dollar for Dollar match up to $50,000 during the 7 PM hour.

Local NBC stations, local radio stations, many local business, and lots of Bay Area foundations will be promoting the Silicon Valley Day of Giving on May 6th. So if MIRI is making a splash with our fundraising that day, it's possible we'll draw attention from media and by extension new donors.

We need your help to make the most of this opportunity!
Making the most of this opportunity will require some cleverness and a lot of coordination. We are going to need all the help we can get. Here are some ways you can help:

We are currently thinking through how to best take advantage of  this unique opportunity. If you have any ideas and/or want to join the planning team (currently Malo, Louie, Luke, and long-time MIRI supporter Alexei Andreev), shoot Malo an email at <EMAIL>.
If you are interested in supporting MIRI with a  donation of $500 or more during the fundraiser, we'd love to coordinate with you to make it count as much as possible. Get in touch with Malo at <EMAIL>.
All MIRI supporters (not just donors) have the potential to make a big impact if we can all work together in a coordinated manner. Sign up below to receive updates on our strategy leading up to the event, and updates throughout the fundraiser on the best times to give and promote the event. Follow along with the excitement and be on the inside of what's going on all day!

The post Help MIRI in a Massive 24-Hour Fundraiser on May 6th appeared first on Machine Intelligence Research Institute.


Roland Siegwart on autonomous mobile robots

 Roland Siegwart (born in 1959) is a Professor for Autonomous Systems and Vice President Research and Corporate Relations at ETH Zurich. After studying mechanics and mechatronics at ETH, he was engaged in starting up a spin-off company, spent ten years as professor for autonomous microsystems at EPFL Lausanne and held visiting positions at Stanford University and NASA Ames.
In his research interests are in the creation and control of intelligent robots operating in complex and highly dynamical environments. Prominent examples are personal and service robots, inspection devices, autonomous micro-aircrafts and walking robots. He is and was the coordinator of European projects, co-founder of half a dozen spin-off companies and board member of various high-tech companies.
Roland Siegwart is member of the Swiss Academy of Engineering Sciences, IEEE Fellow and officer of the International Federation of Robotics Research (IFRR). He is in the editorial board of multiple journals in robotics and was a general chair of several conferences in robotics including IROS 2002, AIM 2007, FSR 2007, ISRR 2009.

Luke Muehlhauser: In 2004 you co-authored Introduction to Autonomous Mobile Robots, which offers tutorials on many of the basic tasks of autonomous mobile robots: locomotion, kinematics, perception, localization, navigation, and planning.
In your estimation, what are the most common approaches to "gluing" these functions together? E.g. are most autonomous mobile robots designed using an agent architecture, or some other kind of architecture?

Roland Siegwart: Mobile robots are very complex systems, that have to operate in real world environments and have to take decisions based on uncertain and only partially available information. In order to do so, the robot's locomotion, perception and navigation system has to be best adapted to the environment and application setting. So robotics is before all a systems engineering task requiring a broad knowledge and creativity. A wrongly chosen sensor setup cannot be compensated by the control algorithms. In my view, the only proven concepts for autonomous decision making with mobile robots are Gaussian Processes and Bayes Filters. They allow to deal with uncertain and partial information in a consistent way and enable learning. Gaussian Processes and Bayes filter can model a large variety of estimation and decision processes and can be implemented in different forms, e.g. as the well-known Kalman Filter estimator.
Most mobile robots use some sort of agent architecture. However, this is not a key issue in mobile robots, but rather an implementation issue for systems that run multiple tasks in parallel. The main perception, navigation and control algorithms have to adapt to unknown situation in a somewhat predictably and consistent manner. Therefore the algorithms and navigation concepts should also allow the robotics engineer to learn from experiments. This is only possible, if navigation, control and decision making is not implemented in a black-box manner, but in a model based approach taking best advantage of prior knowledge and systems models.

Luke: So are you saying that the glue which holds together the perception, navigation, and control algorithms is typically an agent architecture, and this is largely because you need to integrate those functions in a model-based manner which can reveal to the engineer what's going wrong (in early experiments) and how to improve it? Or are you saying something else?

Roland: You understanding is only partially correct. Yes, most robot systems make use of some sort of an agent architecture, because it is the most evident concept to implement independent parallel tasks, like for example robot localization and security stop using the bumper signals. However, I don't see agent architecture as a major issue in robotics or as the main glue. The glue for designing and implementing autonomous robots is with the fundamental understanding of all key elements and its interplay by the robotics engineer. Furthermore, Gaussian Processes and Bayes filter are today the most promising and proven approach for autonomous navigation, especially Simultaneous Localization and Mapping.

Luke: As robotic systems are made increasingly general and capable, do you think a shift in techniques will be required? E.g. 15 years from now do you expect Gaussian Processes and Bayes filters to be even more dominant in robotics than they are today, or do you expect rational agent architectures to ascend, or do you expect hybrid systems control to take over, or what? (Wild speculation is allowed; I know you're not a crystal ball!)

Roland: I consider Gaussian Processes and Bayes filters the most powerful tools to create rational agents. They enable to learn correlations and models, and to reason about situations and future goals. This model-based approaches will gain importance in contrast to behavior-base approaches. However, there will probably never be a single unifying approach for creating intelligent agents.
Robotics is the art of combining sensing, actuation and intelligent control in the most creative and optimal way.

Luke: Why do you expect model-based approaches to gain importance relative to behavior-based approaches?

Roland: In order to take "wise" decisions and plan actions, a robot has to be able to anticipate reactions its decisions and actions might have. This can only be realized by models, that form the basis for predictions. Furthermore, unsupervised learning also requires models that enable the robot system to learn from experience. Models enable the robot to generalize experiences which is not really possible with behavior-based approaches.

Luke: From your perspective, what has been some of the most interesting work in model-based approaches to autonomous robots in the past 5 years?

Roland: I think the most prominent model-based approach in robotics is within SLAM (Simultaneous Localization and Mapping) which can considered to be pretty much solved.
Thanks to consistent application to Gaussian Processes and Bayes filters, and appropriate error modelling, SLAM is today feasible with different sensors (Laser, Vision) and on wheeled and flying platforms.
Large scale maps with considerable dynamics, changes in lightning conditions and loop closures have been demonstrated be groups from Oxford, Sidney University, MIT, ETH an many more.
An other robotics field, where a lot of progress has been achieved by model base approaches, is in imitation learning of complex manipulation task. By combining physical models of human arms and robot manipulators with probabilistic processes, learning of various manipulation task has been demonstrated by groups at USC, DLR, KIT, EPFL and many other places.

Luke: Thanks, Roland!
The post Roland Siegwart on autonomous mobile robots appeared first on Machine Intelligence Research Institute.


Domitilla del Vecchio on hybrid control for autonomous vehicles

 Domitilla Del Vecchio received the Ph. D. degree in Control and Dynamical Systems from the California Institute of Technology, Pasadena, and the Laurea degree in Electrical Engineering from the University of Rome at Tor Vergata in 2005 and 1999, respectively. From 2006 to 2010, she was an Assistant Professor in the Department of Electrical Engineering and Computer Science and in the Center for Computational Medicine and Bioinformatics at the University of Michigan, Ann Arbor. In 2010, she joined the Department of Mechanical Engineering and the Laboratory for Information and Decision Systems (LIDS) at the Massachusetts Institute of Technology (MIT), where she is currently an Associate Professor. She is a recipient of the Donald P. Eckman Award from the American Automatic Control Council (2010), the NSF Career Award (2007), the Crosby Award, University of Michigan (2007), the American Control Conference Best Student Paper Award (2004), and the Bank of Italy Fellowship (2000). Her research interests include analysis and control of networked dynamical systems with application to bio-molecular networks and transportation networks.

Luke Muehlhauser: In Verma & del Vecchio (2011), you and your co-author summarize some recent work in semiautonomous multivehicle safety from the perspective of hybrid systems control. These control systems will "warn the driver about incoming collisions, suggest safe actions, and ultimately take control of the vehicle to prevent an otherwise certain collision."
I'd like to ask about the application of hybrid control to self-driving cars in particular. Presumably, self-driving cars will operate in two modes: "semi-autonomous" (human driver, with the vehicle providing warnings and preventing some actions) and "fully autonomous" (no human driver). Do you think hybrid control will be used for both purposes, in commercial self-driving cars released (e.g.) 10 years from now? Or do you think hybrid control will be competing with other approaches aimed at ensuring safe behavior in autonomous and semi-autonomous vehicles?

Domitilla Del Vecchio: Yes, I believe that hybrid control will be used in commercial vehicles both for autonomous and semi-autonomous functions within the next 10 years. Here is some background to support this belief. The decreasing costs of embedded computing and communication technologies are pushing several of today's engineering systems toward increased levels of autonomy. Transportation systems are an obvious example, in which vehicles and infrastructure are being enriched with more computation, sensing, and communication every day, to increase safety, comfort, and efficiency. For life-critical systems, however, this enrichment raises the fundamental question of whether newly engineered systems with enhanced functionalities can be proven to be safe. A number of accidents have been reported, such as the unintended acceleration problem of a Toyota vehicle in 2008, so that initiatives, including the ISO 26262 for functional safety, have been taken by the auto industry world-wide to address this question. According to these new automotive functional safety standards, any design that affects life critical applications should be assured to be safe. This requirement for system assurance has led most car companies, both in the US and in Europe, to explore formal design and verification approaches so that safety guarantees can be provided on newly designed applications and old applications can be formally verified for safety. In the hybrid control literature, techniques to design mixed logical/dynamical systems under safety specifications have been developed since the 90s with the pioneering work of Tomlin and co-workers and the very well known California PATH project. Therefore, hybrid control approaches have a substantial potential to impact the current automotive technology as far as safety applications are concerned and in fact many companies world-wise are already initiating research programs to explore the promise of formal methods for design and verification. Many challenges, however, need to be overcome from a theoretical point of view, such as being able to handle in a safe and least restrictive way hidden information that arises from many sources, such as driver's behavior, sensor and communication errors, and poorly known environments.

Luke: And do you think hybrid systems control will fill a particular niche in the market for high-assurance software for self-driving cars, or do you think hybrid control approaches will compete with other approaches to high-assurance software, or do you think in the end both hybrid control and other currently existing approaches will need to be replaced by other approaches?

Domitilla: If we intend hybrid control approaches broadly as formal design methodologies that carefully model dynamics and logic, then I do not think they will compete with other approaches but they will complement other approaches. Ultimately, I think a mixture of approaches will be considered, including formal design methods, AI-like methods, and engineering-based methods.

Luke: I've discussed formal design methods in several past interviews. Which kinds of AI-like methods and engineering-based do you expect will be brought to bear on the challenge of high-assurance software for self-driving cars?

Domitilla: There are a number of approaches that are often used in the development of autonomous vehicles mostly for checking collision with static and moving obstacles, such as the RTT and RTT* algorithms that were used in the Urban Grand Challenge by the MIT team, or more general path planning algorithms with obstacles. These are typically originating from the robotics community and are not usually concerned with formal safety guarantees although work often very well in practice. Along the same lines, questions of perception of the environment are obviously crucial in self-driving vehicles, such as recognizing whether something standing on the road side is a pedestrian or a small tree, which may involve different vehicle's decisions for assuring safety. These type of questions have been mostly studied in the artificial intelligence community, including research in computer vision. By engineering-based approaches, I mostly mean the typical software development cycle in industry, which involves extensive testing to highlight possible system malfunctions or safety hazards.

Luke: The hybrid control approach used in Verma & del Vecchio (2011) had, in experiment, a success rate of 96.9%. I presume the success rate will need to be substantially higher before such systems are used to control autonomous cars in real road conditions? How much work might it take to push the success rate up to, say, 99.999%?

Domitilla: That is actually not true for systems that involve human drivers. In fact, automotive companies often prefer to tolerate an epsilon % of collision so they can have less conservative warning/controllers, which override drivers less frequently. From a practical standpoint, if human driver behavior needs to be accounted for, as in Verma and Del Vecchio (2011), 100% safety will most likely not be achievable just because human behavior can be modeled statistically as opposed to non-deterministically. In the paper, we truncated the Gaussian probability distributions that describe how drivers brake/accelerate in the proximity of the intersection in order to have bounded capture sets. Since the tails of the Gaussian probability distributions were not included, we may still have some (rare) instance in which the system cannot guarantee safety. In this sense, I think that when human behavior is accounted for, models and approaches will have to be stochastic and focus on providing probability of safety as opposed to 100%. This is a research direction we are pursuing right now.

Luke: What are the most interesting or important open problems in this area right now, to your mind?

Domitilla: There are many. A couple of ones that are particularly relevant especially from an application standpoint are computational complexity and the ability to provide design methods for stochastic safety. The first one is a problem that is always a challenge for implementing most formal safety approaches. Algorithms usually do not scale well with the size of the system and this limits real-time applicability. The second one is a challenging technical problem and methods to efficiently handle hidden decision making in a least conservative manner are very much needed.

Luke: What are your recommended readings on the latter challenge?

Domitilla: There are many readings that address several of the aspects involved in this problem. In the control theory community, we have seen recent papers by Abate's and Lygeros' groups, which address the stochastic reachability/verification problem. In the robotics community, there are also very related works, in which a vehicle has to be controlled to avoid stochastically moving agents, which can move according to a set of behaviors, with a given least probability (Jon How and colleagues). I believe there are a few more, and the ones cited here are not exhaustive, but these are those that come to mind now.

Luke: Thanks, Domitilla!
The post Domitilla del Vecchio on hybrid control for autonomous vehicles appeared first on Machine Intelligence Research Institute.