How Big is the Field of Artificial Intelligence? (initial findings)

Co-authored with Jonah Sinick.
How big is the field of AI, and how big was it in the past?
This question is relevant to several issues in AGI safety strategy. To name just two examples:

AI forecasting. Some people forecast AI progress by looking at how much has been accomplished for each calendar year of research. But as inputs to AI progress, (1) AI funding, (2) quality-adjusted researcher years (QARYs), and (3) computing power are more relevant than calendar years.1 To use these metrics to predict future AI progress, we need to know how many dollars and QARYs and computing cycles at various times in the past have been required to produce the observed progress in AI thus far.
Leverage points. If most AI research funding comes from relatively few funders, or if most research is produced by relatively few research groups, then these may represent high-value leverage points through which one might influence the field as a whole, e.g. to be more concerned with the long-term social consequences of AI.

For these reasons and more, MIRI recently investigated the current size and past growth of the AI field. This blog post summarizes our initial findings, which are meant to provide a "quick and dirty" launchpad for future, more thorough research into the topic.
To begin, we tried to quantify the size and past growth of the field using metrics such as

Number of researchers
Number of journals
Publication counts
Number of conferences
Number of organizations
Famous prizes awarded for AI research
Amount of funding

It's difficult to interpret these figures, and they may be significantly less informative than an object level study of the research would be, but the figures still have some relevance:

For the purpose of investigating growth, one can look at year-to-year percentage growth in the statistics, combining this with other measures of the amount of progress that has occurred in AI, in order to estimate the amount of AI research that will occur in the medium term future.
For the purpose of investigating the current size of the AI field, one can look at the quantitative metrics relative to the corresponding metrics for computer science (CS)2 , and use these in conjunction with a holistic sense of the current size of the CS field to inform one's holistic sense of the amount of progress that there's been in AI.

The data that we were able to collect provide a decent picture of the size of the AI field relative to the size of the CS field, but they are insufficient to support a robust conclusion, and more investigation is warranted. Unless otherwise specified, see the spreadsheet "Current size & past growth of AI field" for the raw data on which this blog post is based.
The size of the AI field
According to a variety of metrics, the amount of AI research being done appears to be about 10% of the amount of computer science (CS) research being done. The metrics used, however, mostly capture research quantity rather than research quality, and thus may be a weak proxy for measuring how many QARYs have been invested. That said, the fact that roughly 10% of CS research prizes are awarded for AI work may indicate that research quality is similar in CS and AI.
We obtained many of the relevant figures from Microsoft Academic Search (MAS). MAS allows one to search under the headings:

Computer science
Artificial intelligence
Natural language and speech
Machine learning and pattern recognition
Computer vision

One gets different figures depending on whether one counts the latter three subjects (hereafter referred to as "cognate disciplines") as AI. Below, we give figures both for items that fall under the "artificial intelligence" heading alone, and for items that fall under the heading "artificial intelligence" or under the heading of one of the cognate disciplines.
Number of researchers
MAS gives number of authors in CS, AI, and the cognate disciplines of AI, but these figures don't pick up on the amount of research done as well as publication count figures do.3
The IEEE Computational Intelligence Society has ~7,000 members and the IEEE Computer Society has ~85,000 members, so the membership of the first is 8% the membership of the second.
Some other relevant figures (which don't paint a cohesive picture):

According to the Bureau of Labor Statistics, there are 26,700 computer and information science researchers in the US.
ACM's Special Interest Group on Artificial Intelligence (SIGAI) has "more than 1,000 members."
The International Neural Network Society (INNS) has "more than 2,000 members."

Number of journals
MAS lists 1360 CS journals, with 106 in AI, and 172 in either AI or one of AI's cognate disciplines, so 8% and 13% respectively.4
Publication counts
Between 2005 and 2010, of those publications listed under MAS's "CS" heading, about 10% were listed under "AI" and about 20% were listed under "AI" or one of its cognate disciplines.5 One sees roughly the same percentages if one looks at publications between 1990 and 1995, between 1995 and 2000, and between 2000 and 2005.6 Searching Google Scholar for "Computer Science" and "Artificial Intelligence," one finds that the number of hits for the latter search is about 30% the number of hits for the former search,7 which could mean that the amount of AI research is significantly more than 10% the amount of CS research, but some papers that contain the phrase "artificial intelligence" are not artificial intelligence research, and some computer science papers may not contain the phrase "computer science."
Number of conferences
MAS lists 3,519 "top conferences" in CS and 361 "top conferences" in AI, and the former number is about 10% of the latter number. There are 561 "top conferences" in AI or cognate disciplines, so 16% the number of CS conferences.8
Number of organizations
Microsoft Academic Search lists 11,338 organizations for CS and 7,125 organizations for AI, so 63%. If one counts cognate disciplines as AI, the number of AI organizations is 21,802, so 192% that of CS organizations.9 Taken in isolation, this would suggest that the amount of AI research is much greater than 10%.
"Number of organizations" seems likely to be a weaker metric of amount of research than "number of publications," etc., so this should be discounted. Nevertheless, the fact that the ratio of AI organizations to CS organizations is so much higher than the other ratios that we looked at is a puzzle. Perhaps the difference comes from the CS community and the AI community having different cultural norms. Or, perhaps MAS is less consistent about how it counts organizations than how it counts publications.
Famous prizes awarded for AI research vs. CS research
ACM Turing Award: Six out of 46 prizes were awarded for AI research, so 13% of the total.10
Nevanlinna Prize: One of the 8 prizes was awarded for AI work, so 12.5% of the total. However, the prize for AI work was awarded in 1986, which is a long time ago.11
Amount of funding
In 2011, the National Science Foundation (NSF) received $636 million for funding CS research (through CISE). Of this, $169 million went to Information and Intelligent Systems (IIS). IIS has three programs: Cyber-Human Systems (CHS), Information Integration and Informatics (III) and Robust Intelligence (RI). If roughly 1/3 of the funding went to each of these, then $56 million went to Robust Intelligence, so 9% of the total CS funding. (Some CISE funding may have gone to AI work outside of IIS — that is, via ACI, CCF, or CNS — but at a glance, non-IIS AI funding through CISE looks negligible.)
Other major U.S. funding sources for CS research include ONR, DARPA, and several companies (Microsoft, Google, IBM, etc.) but we have not investigated these funding sources yet. We also did not investigate non-U.S. funding sources.
The growth of the AI field
We did not investigate the growth rate of the number of AI researchers in sufficient depth to make meaningful estimates. However, the growth rate of the number of scientists and engineers in all fields might serve as a very weak proxy measure for the growth rates of AI or CS.
For example, the annual growth rate of science and engineering researchers in OECD countries, between 1995 and 2005, appears to be about 3.3%, corresponding to a doubling time of 23 years.12 This needs to be viewed in juxtaposition with indications that average researcher productivity (as measured by patents per researcher, amount of time spent training per researcher, the number of coauthors per paper, and the number of papers cited) has been decreasing.13 The NSF Budget for Information and Intelligent Systems (IIS) has generally increased between 4% and 20% per year since 1996, with a one-time percentage boost of 60% in 2003, for a total increase of 530% over the 15 year period between 1996 and 2011.14 "Robust Intelligence" is one of three program areas covered by this budget. According to MAS, the number of publications in AI grew by 100+% every 5 years between 1965 and 1995, but between 1995 and 2010 it has been growing by about 50% every 5 years. One sees a similar trend in machine learning and pattern recognition.15
Notes on further research
Future research on this topic could dig much deeper, and come to more robust conclusions. Our purpose here is to lay some groundwork for future research. With that in mind, here are some miscellaneous notes to future researchers investigating the current size and past growth of the AI field:

If the papers being cited are newer, that could indicate more rapid progress. On the other hand, it could also indicate faddishness, and one would somehow need to differentiate between the two things.
Some citation databases that could be useful for analyzing citation patterns are are Scopus, Web of Science, MS Academic Search, and Science Citation Index (SSI).16
Some sources of noise in citation counts are: (a) Journal editors asking authors of submitted papers to add citations to other papers in the same journal in order to boost the journal's impact factor & (b) Authors citing their own papers excessively in order to increase their citation counts.17

Our thanks to Sebastian Nickel for data-gathering, and to Carl Shulman for his feedback.
Another important input metric is theoretical progress imported from other fields, e.g. methods from statistics.It's also worth noting the following point. Suppose that a source S can be used to generate an estimate E1 for a quantity Q1 having to do with AI and an estimate E2 having to do with CS. Then E1 and E2 may overstate or understate Q1 and Q2 (respectively). Let the factors by which E1 and E2 differ from Q1 and Q2 be F1 and F2. We don't have good estimates for F1 and F2, but if we compute the ratio 1(E1)/(E2) we get [(Q1)/(Q2)]*[(F1)/(F2)]. The quantity (F1)/(F2) will be closer to 1 than F1 is to 1, because the some of the factors that lead E1 to deviate from Q1 to given degrees will also lead E2 to deviate from Q2 to similar degrees. So (E1)/(E2) is closer to (Q1)/(Q2) (in relative terms) than E1 is to Q1 (in relative terms).MAS shows 1.6 million authors in CS and 0.26 authors million in AI, so 16%. If one adds up the listed number of authors in AI and cognate disciplines, the figure rises to 39%. However some authors publish in multiple disciplines (for example, an author might publish in both artificial intelligence and machine learning).Cells B96 through B100 of the spreadsheet.Some papers may be listed under multiple categories, making it unclear whether the 10% figure or the 20% figure is more representative.Table with upper left hand corner A2 in the spreadsheet.Google Scholar results:
Search term "computer science" (in quotes) yields 2,650,000 results
"artificial intelligence" -> 1,710,000
"machine intelligence" -> 655,000
since 2013:
Search term "computer science" (in quotes) yields 99,600 results
"artificial intelligence" -> 32,300
"machine intelligence" -> 11,600
2012:
Search term "computer science" (in quotes) yields 163,000 results
"artificial intelligence" -> 52,500
"machine intelligence" -> 22,600
2011:
Search term "computer science" (in quotes) yields 247,000 results
"artificial intelligence" -> 66,100
"machine intelligence" -> 23,000Cell B27 through Cell B31 of the spreadsheet.Cell B119 through Cell B123 of the spreadsheet.The annual Turing Prize was first awarded in 1966 (last prize 2012), so 46 prizes so far. Of those, 6 were for achievements in AI related research, namely:
• 1969 Marvin Minsky
• 1971 John McCarthy
• 1975 Newell & Simon
• 1991 Robin Milner (machine assisted proof construction)
• 1994 Edward Feigenbaum & Raj Reddy
• 2010 Leslie Valiant (Probably Approximately Correct Learning)
• 2011 Judea Pearl
8 of the 46 prizes were awarded to 2 people, and another 2 were awarded to 3 people, so the total number of recipients is 58, out of which 8 received the prize for AI-related achievements.The Nevanlinna Prize has been awarded every 4 years since 1982; 8 times so far.From U.S. National Science Foundation (NSF). Science and Engineering Indicators: 2010, Chapter 3. Science and Engineering Labor Force:
In the early 1960s, a prominent historian of science, Derek J. de Solla Price, examined the growth of science and the number of scientists over very long periods in history and summarized his findings in a book entitled Science Since Babylon (1961). Using a number of empirical measures (most over at least 300 years), Price found that science, and the number of scientists, tended to double about every 15 years, with measures of higher quality science and scientists tending to grow slower (doubling every 20 years) and measures of lower quality science and scientists tending to grow faster (every 10 years). According to Price (1961), one implication of this long-term exponential growth is that "80 to 90% of all the scientists that ever lived are alive today." This insight follows from the likelihood that most of the scientists from the past 45 years (a period of three doublings) would still be alive. Price was interested in many implications of these growth patterns, but in particular, he was interested in the idea that this growth could not continue indefinitely and the number of scientists would reach "saturation." Price was concerned in 1961 that saturation had already begun.
How different are the growth rates in the number of scientists and engineers in recent periods from what Price estimated for past centuries? Table 3-A shows growth rates for some measurements of the S&E labor force in the United States and elsewhere in the world for a period of available data. Of these measures, the number of S&E doctorate holders in the United States labor force showed the lowest average annual growth of 2.4% (doubling in 31 years if this growth rate were to continue). The number of doctorate holders employed in S&E occupations in the United States showed a faster average annual growth of 3.8% (doubling in 20 years if continued). There are no global counts of individuals in S&E, but counts of "researchers" in member countries of the Organisation for Economic Co-operation and Development (OECD) grew at an average annual rate of 3.3% (doubling in 23 years if continued). Data on the population of scientists and engineers in most developing countries are very limited, but OECD data for researchers in China show a 10.8% average annual growth rate (doubling in 8 years if continued). All these numbers are broadly consistent with a continuation of growth in S&E labor exceeding the rate of growth in the general labor force.Below are some references on declining productivity per researcher. Our thanks to Gwern for compiling many of these in the article Scientific Stagnation:
• Machlup, Fritz. The Production and Distribution of Knowledge in the United States, Princeton, NJ: Princeton University Press, 1962, 170-176
• Segerstrom, Paul. Endogenous Growth Without Scale Effects, American Economic Review, December 1998, 88, <PHONE>
• Terman, F.E. A Brief History of Electrical Engineering Education, Proceedings of the IEEE, August 1998, 86 (8), <PHONE>
• Adams, James D., Black, Grant C., Clemmons, J.R., and Stephan, Paula E. Scientific Teams and Institutional Collaborations: Evidence from U.S. Universities, <PHONE>, NBER Working Paper #10640, July 2004
• Jones (2006), Age and Great Invention
• Jones, Benjamin F. The Burden of Knowledge and the Death of the Renaissance Man: Is Innovation Getting Harder? NBER Working Paper #11360, 2005
• National Research Council, On Time to the Doctorate: A Study of the Lengthening Time to Completion for Doctorates in Science and Engineering, Washington, DC: National Academy Press, 1990
Tilghman, Shirley (chair) et al. Trends in the Early Careers of Life Sciences, Washington, DC: National Academy Press, 1998
• Zuckerman, Harriet and Merton, Robert. Age, Aging, and Age Structure in Science, in Merton, Robert, The Sociology of Science, Chicago, IL: University of Chicago Press, 1973, 497-559
• Cronin et al, 2004 Visible, Less Visible, and Invisible Work: Patterns of Collaboration in 20th Century Chemistry, Journal of the American Society for Information Science and Technology, 2004, 55(2), 160-168
• Grossman, Jerry. The Evolution of the Mathematical Research Collaboration Graph, Congressus Numerantium, 2002, 158, 202-212
• Cronin, Blaise, Shaw, Debora, and La Barre, Kathryn. A Cast of Thousands: Coauthorship and Subauthorship Collaboration in the 20th Century as Manifested in the Scholarly Journal Literature of Psychology and Philosophy, Journal of the American Society for Information Science and Technology, 2003, 54(9), 855-871
• McDowell, John, and Melvin, Michael. The Determinants of Coauthorship: An Analysis of the Economics Literature, Review of Economics and Statistics, February 1983, 65, 155-160
• Hudson, John. Trends in Multi-Authored Papers in Economics, Journal of Economic Perspectives, Summer 1996, 10, 153-158
• Laband, David and Tollison, Robert. Intellectual Collaboration, Journal of Political Economy, June 2000, 108, 632-662
• Jones 2010. As Science Evolves, How Can Science Policy?
• The Collapse of the Soviet Union and the Productivity of American Mathematicians, by George J. Borjas and Kirk B. Doran, NBER Working Paper No. 17800, February 2012
See table with upper left-hand corner A367 in the spreadsheet.Table with upper left hand corner A2 in the spreadsheet.A 2008 study compared PubMed, Scopus, Web of Science, and Google Scholar and concluded: "PubMed and Google Scholar are accessed for free […] Scopus offers about 20% more coverage than Web of Science, whereas Google Scholar offers results of inconsistent accuracy. PubMed remains an optimal tool in biomedical electronic research. Scopus covers a wider journal range […] but it is currently limited to recent articles (published after 1995) compared with Web of Science. Google Scholar, as for the Web in general, can help in the retrieval of even the most obscure information but its use is marred by inadequate, less often updated, citation information." Larsen & von Ins (2010) claim that the coverage of SSI has been declining.Here are some caveats about citations as a measure of quality: Wilhite and Fong (2012): "…impact factors continue to be a primary means by which academics "quantify the quality of science". One side effect of impact factors is the incentive they create for editors to coerce authors to add citations to their journal. Coercive self-citation does not refer to the normal citation directions, given during a peer- review process, meant to improve a paper. Coercive self-citation refers to requests that (i) give no indication that the manuscript was lacking in attribution; (ii) make no suggestion as to specific articles, authors, or a body of work requiring review; and (iii) only guide authors to add citations from the editor's journal." And Storbeck (2012): "The [extent] of manipulation is amazing. For example, according to figures published by the Managing Editor of the 'Review of Finance', the impact factor of the 'Journal of Banking and Finance' – the fourth worst offender according to the study by Wilhite and Fong – dwindles if self-citations are excluded. While the raw impact factor of the journal is 2.731, the one without self-citations is just 0.748."The post How Big is the Field of Artificial Intelligence? (initial findings) appeared first on Machine Intelligence Research Institute.


Existential Risk Strategy Conversation with Holden Karnofsky

On January 16th, 2014, MIRI met with Holden Karnofsky to discuss existential risk strategy. The participants were:

Eliezer Yudkowsky (research fellow at MIRI)
Luke Muehlhauser (executive director at MIRI)
Holden Karnofsky (co-CEO at GiveWell)

We recorded and transcribed the conversation, and then edited and paraphrased the transcript for clarity, conciseness, and to protect the privacy of some content. The resulting edited transcript is available in full here (41 pages).
Below is a summary of the conversation written by Karnofsky, then edited by Muehlhauser and Yudkowsky. Below the summary are some highlights from the conversation chosen by Karnofsky.
See also three previous conversations between MIRI and Holden Karnofsky: on MIRI strategy, on transparent research analyses, and on flow-through effects.

Summary
Pages 1-5:
Defining some basic terms, such as the hypothesis that far future considerations dominate utility (the "N lives" hypothesis). Holden temporarily accepts this hypothesis for the sake of argument.

Pages 5-9:
Discussion of how the path to colonization of the stars is likely to play out.
There has been significant confusion over the definition of the term "existential risk." Going into the conversation, Holden interpreted this term to mean "risks of events that would directly and more-or-less immediately wipe out the human race," whereas Eliezer and Luke meant to use it to mean anything that causes humanity to fall drastically short of its potential, including paths that simply involve never succeeding in e.g. space colonization. This confusion took up much of pages 1-9 and was clarified at pages 8-9.

Pages 9-13:
Further discussion of how the path to colonization of the stars is likely to play out. There isn't much in the way of clear and major disagreement except for the dynamics around artificial general intelligence.

Pages 14-19:
Holden argues for an emphasis on "doing good without worrying much about its connection to the far future" as plausibly the best way to improve the far future. At the same time, he thinks it's likely that global catastrophic risks are a promising area for philanthropy and thinks it's likely that GiveWell will make recommendations relevant to these.
Holden thinks that many of the arguments advanced in favor of focusing on global catastrophic risks are "right for the wrong reasons," focusing on Pascal's Mugging type arguments and not sufficiently addressing the question of whether these risks are neglected, and how neglected compared to other areas. Eliezer and Luke agree that addressing this question is crucial (though they are confident that the areas they emphasize as promising are sufficiently neglected to justify this).
Holden wants to do more investigation into what's neglected before being confident that these areas are relatively neglected, but his current guess is that they are. He discusses to some extent why GiveWell hasn't focused on them previously despite suspecting this.

Pages 19-27:
Holden thinks that donating to AMF (a) may make the world more robust to catastrophe in multiple ways, if in fact "robust to catastrophe" turns out to be the most important criterion; (b) is a good thing to do if that criterion does not turn out to be so important and if our ability to identify key risk factors for the far future is essentially nil; (c) has helped GiveWell along its path to becoming influential and able to work on other fronts. He believes that the amount of good done on the (a) front is small relative to the amount of influence over the future of humanity that people like Eliezer and Luke seek to have, and that people who think they can have a great deal of influence on the future of humanity have reason to pursue other opportunities, but that it isn't obvious that a donor (especially the sort of individual donor GiveWell has historically targeted) can do as much good as Luke and Eliezer seem to aim for.
Eliezer and Luke agree that doing "normal good stuff" has a good track record and is likely to produce significant goods in the near term, but they worry that faster economic growth accelerates AI capabilities research relative to AI safety/Friendliness research, since the former parallelizes more easily across multiple researchers and funding sources than the latter does. If smarter-than-human AI is figured out before Friendliness is, they expect global catastrophe. Hence, faster economic growth could be net bad in the long run. But, they're much, much less sure of the sign of this argument than they are about e.g. the intelligence explosion and the orthogonality of capabilities and values.

Pages 28-32:
It's possible that the disagreements between Holden and Luke/Eliezer reduce to their disagreements over the specific dynamics of AI risk. Holden feels that the "Astronomical Waste" essay has been misleadingly cited, as establishing that one should directly focus on x-risk reduction work over other causes; Eliezer feels the essay didn't say anything incorrect but has been misinterpreted on this point.

Pages 32-35:
Discussion of how to deal with a situation in which there are multiple models of the world that might be right, and some involve much higher claimed utilities than others. Does one do a straightforward expected-value calculation, which means essentially letting the model with the bigger numbers in it dominate? Or does one do something closer to "giving each model votes/resources in proportion to their likelihood"? Every principled/explicit answer seems to have problems. Holden, Luke and Eliezer agree that the "multiply big utilities by small probabilities" approach is not optimal.

Pages 35-39:
Discussion of likelihood that humanity will eventually colonize the stars. Luke and Eliezer think that this is a default outcome if things continue to go moderately well; Holden had the impression that this was not the case, but hasn't thought about it much. Possibilities for updating on this question are discussed.

Pages 39-45:
Discussion of whether "causing N future lives to exist when they would have otherwise not existed" should be valued at least 10%*N times as much as "saving one life today." Holden has high uncertainty on this question, more so than Eliezer and Luke. He mistrusts the methodology of thought experiments to a greater degree than Eliezer and Luke.

Some highlights selected by Holden
Holden: Yeah, I agree. I think you want a list of all the things that could be highly disruptive and you want to consider them all risks, and you want to consider them all possibilities, I'm not really sure what else there is here.
Luke: I think we might also disagree on what you can figure out, and what you can't, about the future.
Holden: Yeah, I think that's our main disagreement.
Luke: Because I think we make a list and we think we know some things about the items on that list and therefore we can figure out which ones to focus on more.
Holden: Well, no, I would agree with what you just stated, as stated, but I just think that you are more confident than I am. I also believe we can make a list of things that are much more likely to be disruptive than other things, and then we should go and look into them and pay attention to them, but I just think that you guys are much more confident in your view of what are the things. My feeling is this: my feeling is basically we know very little. It's very important to keep this [upward trend] going. That is not something we should neglect or ignore. So generally helping people is good, that's kind of how we've gotten to where we've gotten, is that people have just done things that are good, without visualizing where they're going.
The track record of visualizing where it's all going is really bad. The track record of doing something good is really good. So I think we should do good things, I also think that we should list things that we think are in the far future or just relevant to the far future that are especially important. I think we should look into all of them. Another point worth noting is that my job is different from you guy's job. You guys are working in an organization that's trying to … it's a specialized organization, it's knowledge production. My job is explicitly to be broad. My job is to basically be able to advise a philanthropist and part of what I want to be able to do is to be able to talk about a lot of different options and know about a lot of different things. I don't think it's a good way for me to do my job, to just pick the highest expected value thing and put all my eggs in that basket. But perhaps that would be a good job for many people, just not for someone whose explicit value-add is breadth of knowledge.
So part of it is the role, but I do think that you guys are much more confident. My view is that we should list things we think we know, we should look into doing something about them. At the same time, we should also just do things that are good, because doing things that are good has a better track record of getting us closer to colonizing the stars than doing things that are highly planned out.
Eliezer: So indeed, if I tried to pass your ideological Turing test, I would have said some mixture of "we can't actually model the weird stuff and people trying to do good is what got us where we are and it will probably take us to the galaxy as well," that would have been the very thing …
Holden: You just need to water down a little.
Eliezer: Sure, so: "insofar as we're likely to get to the galaxy at all, and it's highly probable that a lot of the pathway will be people just trying to do good, so just try to do good and get there."
Holden: Yeah, and it especially will come from people just doing good, as approximate goal and then having kind of civilizational consequences in ways that were hard to foresee, which is I'm particularly interested in opportunities to do good that just feel big. Even if the definition of big is different from opportunity to opportunity, so like a way to help a lot of animals. A way to help a lot of Africans, a way to help a lot of Americans. These are all, in some absolute sense, it seems unlikely that they could be in the same range, but in some market efficiency sense, they're in the same range. This is: whoa, I don't see something this good every day, most things is good someone else snaps up, let me grab this one, because this is the kind of thing that could be like a steam engine, where it's like, this thing is cool, I built it. it's super cool. Then it actually has civilizational consequences.
Eliezer: So in order to get an idea of what you think Earth's ideal allocation of resources should be, if you were appointed economic czar of the world, you formed the sort of dangerous to think about counterfactual… or maybe a better way of putting it would be: how much money would you need to have personal control over before you started to trying to fund, say, bioterror, nanotech and AI risk type stuff? Not necessarily any current organization, but before you start trying to do something about it?
Holden: I mean, less than Good Ventures has.
Eliezer: Interesting.
Holden: Like, I think we're probably going to do something. But it depends. We want to keep looking into it. Part of this is that I don't have all the information and you guys may have information that I don't, but I think a lot of people in your community don't have information and are following a Pascal's mugging-type argument to a conclusion that they should have just been a lot more critical of, and a lot more interested in investigating the world about. So my answer is: we're still looking at all this stuff, but my view is that no, existential risks are a great opportunity. There's not nearly enough funding there.

***
Holden: So anyway, that was an aside. I think you guys are more in the camp of thinking you understand the issues really well and not only understanding what the issues are, but who is working on what and believing that the neglectedness of x-risk is a large part of your interest in x-risk, I think. I think there are a lot of people who reason so quickly to believing x-risk is paramount that I don't believe they've gone out and looked at the world and seen what is neglected and what isn't neglected. I think they're instead doing a version of Pascal's mugging. But I'm happy to engage with you guys and just say that I don't know everything about what's neglected and what isn't, I think existential risk looks pretty neglected, preliminarily, but I want to look at more things before I really decide how neglected it is and what else might be more neglected. Do you agree with me that the neglectedness of x-risk is a major piece of why you think it's a good thing to work on?
Luke: I think it is for me.
Eliezer: I think I would like specialize that to say that there are particular large x-risks that look very neglected, which means you get a big marginal leverage from acting on them. But even that wouldn't really honestly actually carry the argument.
Holden: But if you read "Astronomical Waste," it concludes that x-risk is the thing to work on, without discussing whether it's neglected, and I think that's the chain of reasoning most people are following. I think that is screwed up.
Eliezer: Yeah, that can't possibly be right. Or a sane Earth has some kind of allocation across all philanthropies. And insofar as things drop below their allocations, you'll get benefit from putting stuff into them, and if they go above their allocations, you're betting off putting your money somewhere else. There exists some amount of investment we can make in x-risk, such that our next investment should be in Against Malaria Foundation or something. Although that actually that still isn't right, because that's a better argument now because GiveWell actually did say Against Malaria Foundation is temporarily overinvested, let's see what they can do what with their existing inflow.
Luke: Though not necessarily relative to the current allocation in the world!
Holden: Yeah, absolutely.

***
Holden: Yeah. Okay. I think some disagreements we have, which I think are like not enormous disagreements, I think they mostly have to do with how confident we can be. I think we agree that there are many things that are important, we agree that being neglected is part of what makes a cause good. If there are other causes that are really important and really neglected, those are good, too. We agree that everything that is good has some value, but I think the things that are good have more value relative to the things that seem to fit into the long-term plan and that has a lot to do with my feeling about how confident we can be about the long-term plan.
Eliezer: My reasoning for CFAR sounds a lot like this. Why, to some extent, I sort in practice, divide my efforts between MIRI and CFAR, is sort of like this, except that no matter what happens, I expect the causal pathways to galactic colonization to go down the "something weird happens and other weird things potentially prevent you from doing it" path.
I think that human colonization of the galaxy has probability nearly zero.
Holden: Right, you think it would be something human-like.
Eliezer: I'm hoping that they're having fun and that they have this big, complicated civilization and that there was sort of a continuous inheritance from human values, because I think fun is at present, a concept that exists among humans and maybe to some lesser extent, other mammals, but not in the rocks. So you don't get it for free, you don't want a scenario where the galaxies end up being turned into paperclips or something. But: "humane" life might be a better term.
Holden: Sure, sure.
Eliezer: I think that along the way there you get weird stuff happening and weird emergencies. So CFAR can be thought of as a sort of generalized weird emergencies handler.
Holden: There's a lot of generalized weird emergencies handlers.
Luke: Yeah, you can improve decision-making processes in the world in general, by getting prediction market standard or something.
Holden: Also just by making people wealthier and happier.
Eliezer: Prediction markers have a bit of trouble with x-risks for obvious reasons, like the market can't pay off in most of the interesting scenarios.
Holden: I think you can make humanity smarter by making it wealthier and happier. It certainly seems to be what's happened so far.
Eliezer: Yeah, and intelligence enhancement?
Holden: Yeah, well, that, too. But that's further off and that's more specific and that's more speculative. I think the world really does get smarter, as an ecosystem. I don't mean the average IQ. I think the ecosystem gets smarter. If you believe that MIRI is so important, I think the existence of MIRI is a testament to this, because I think the existence of MIRI is made possible by a lot of this wealth and economic development, certainly it's true for GiveWell. If you take my egg and run it back 20 years, my odds of being able to do anything like this are just so much lower.
Eliezer: CFAR, from my perspective, it's sort of like: generalize those kind of skills required to handle a weird emergency like MIRI and have them around for whatever other weird stuff happens.
Holden: I think the world ecosystem has been getting better in handling weird emergencies like that. I think that part of that, if you want to put a lot of weight on your CFARs, then I think that's evidence, and if you don't want to put a lot of weight, then I think there's other evidence. There is more nonprofits that deal with random stuff, because we have more money.
Eliezer: I'm not sure if I'd rate our ability to handle weird emergencies as having increased. Nuclear weapons are the sort of classic weird emergency that actually did get handled by this lone genius figure who saw it coming and tried to mobilize efforts and so on, I'm talking about Leó Szilárd. So there was a letter to President Roosevelt, which Einstein wrote, except Einstein didn't write it. It was ghost-written by someone who did it because Leó Szilárd told them to, and then Einstein sent it off. There is this sort of famous story about the conversation where Leó Szilárd explains to Einstein about the critical fission chain reaction and Einstein sort of goes "I never thought of that." Then came the Manhattan Project, which was this big mobilization of government effort to handle it.
So my impression is that if something like that happened again, modern day Einstein's letter does not get read by Obama. My impression is that we've somehow gotten worse at this.
Holden: I don't agree with that.
Luke: Eliezer, why do you think that?
Holden: You're also pointing to a very specific pathway. I'm also thinking about all the institutions that exist to deal with random stuff these days. And all the people who have the intellectual freedom, the financial freedom, to think about this stuff, and not just this stuff, other stuff that we aren't thinking about, that can turn out to be more important.
Eliezer: We don't seem to be doing very well with sort of demobilizing the nuclear weapons of the former Soviet Republics, for example.
Holden: We're also talking about random response to random stuff. I think we just have a greater degree of society's ability to notice random stuff and to think about random stuff.
Eliezer: That's totally what I would expect on priors, I'm just wondering if we can actually see evidence that it's true. On priors, I agree that that's totally expected.
Eliezer: We could somehow be worse than it was in the 1940s and yet, still, increasing development could all else equal improve our capacity to handle weird stuff. I think I'd agree with that. I think that I would like also sort of agree that all else being equal, as society becomes wealthier, there are more nonprofits, there is like more room to handle weird stuff.
Holden: Yeah, it's also true that as we solve more problems, people go down the list, so I think if it hadn't been for all the health problems in Africa, Bill Gates might be working on [GCRs], or he might be working on something else with global civilizational consequences. So when I'm sitting here not knowing what to do and not feeling very educated in the various speculative areas, but knowing that I can save some lives, that's another reason there is something to that.
But it's certainly like: the case for donating to AMF, aside from the way in which it helps GiveWell, is definitely in a world in which I feel very not very powerful and not very important [relative to the world Eliezer and Luke envision]. I feel like, you know, I'm going to do [a relatively small amount of] good and that's what I'm trying to do.
So in some sense, when you say, AMF isn't like a player in the story or something, I think that's completely fair, but also by trying to take a lot of donors who are trying to do this much [a small amount] and trying to help them, we've hopefully gotten ourselves in a position to also be a player in the story, if in fact the concept of a player in the story ends up making sense. If it doesn't and this [small amount of good] turns out to be really good, we'll at least have done that.
Eliezer: The sort of obvious thing that I might expect Holden to believe, but I'm not sure that that actually passes your ideological Turing test is that collectively, fixing this stuff collectively, is like a bigger player than collectively the people who go off and try to fix weird things that they think that the fate of the future will hinge on.
Holden: I just think it's possible that what you just said is true, and possible that it isn't. If I'm sitting here, knowing very little about anything, and I want to do a little bit of good, I think doing a little bit of good is better than taking a wild guess on something that I feel very ill-informed about. On the other hand, our ideal at GiveWell is to really be playing both sides.

***
Eliezer: Do we think a necessary and sufficient cause of our disagreement is just our visualization of how AI plays out?
Holden: I think it's possible.
Eliezer: If your visualization of how AI worked magically, instantly switched to Eliezer Yudkowsky's visualization of how AI worked. I mean, Eliezer Yudkowsky, given sudden magical control of GiveWell, does not just GiveWell to be all about x-risk. Eliezer puts it on the link like three steps deep, and just sort of tries to increase the degree to which incoming effective altruists are funneled toward…
…
Holden: I think it's pretty possible, and I just want to contrast what you guys think with the normal tenor of the arguments I have over x-risk, which… I just talk to a lot of people who are just like, look, x-risk is clearly the most important thing. Why do you think that? Well, have you read "Astronomical Waste?" Well, that's a little bit absurd. You have an essay that doesn't address whether something is neglected, concludes what's most important, and we're not even talking about AI and path to AI and why AI, it's just x-risk, [which people interpret to mean things like] asteroids, come on.
Eliezer: I endorse your objection. We can maybe issue some kind of joint statement, if you want, to inform people.
Holden: Yeah, perhaps. I was going to write something about this, so maybe I'll run it by you guys. To the extent that I'm known as Mr. X-Risk Troll, or whatever, it's because those are the arguments I'm always having. When I think about you guys, I think that you and I do not see eye to eye on AI, and that goes back to that conversation we had last time, and that may be a lot of the explanation. At the same time, it's certainly on the table for us to put some resources into this.

***
Eliezer: Okay, I checked the "Astronomical Waste" paper and everything in there seemed correct, but I can see how we would all now wish, in retrospect, that a caveat had been added along the lines of "and in the debate over what to do nowadays, this doesn't mean that explicit x-risk focused charities are the best way to maximize the probability of okay outcome."
Holden: Right, and in fact, this doesn't tell us very much. This may prove a useful framework, it may prove a useless framework. There's many things that have been left unanswered, whereas the essay really had a conclusion of: we've narrowed it down from a lot to a little.
Eliezer: I don't remember that being in that essay. It was just sort of like, this is the criterion by which we should choose between actions, which seems like obviously correct in my own ethical framework.
Holden: I also don't agree with that, so maybe that's the next topic.
Eliezer: Yeah. Suppose that you accepted Maximized Probability of Okay Outcome, not as a causal model of how the world works, but just as a sort of a determining ethical criterion. Would anything you're doing change?
Holden: I've thought about this, maybe not as hard as I should. I don't think much would change. I think I would be relatively less interested in direct, short-term suffering stuff. But I'm not sure by a lot. Actually, I think I would be substantially now. I think five years ago, I wouldn't have changed much. I think right now I would be, because I feel like we're becoming better positioned to actually target things, I think I would be a little bit more confident about zeroing in on extreme AI and the far future and all that stuff. And the things that I think matter most to that, but I don't think it would be a huge change.
***
Holden: I just think there's also a chance that this whole argument is crap and… so there is one guy [at GiveWell] who is definitely representing more the view that we're not going to have any causal impact on all this [far future] stuff and there is suffering going on right now and we should deal with it, and I place some weight on that view. I don't do it the way that you would do it in an expected value framework, where it's like according to this guy, we can save N lives and according to this guy, we could save Q lives and they have very different world models. So therefore, the guy saying N lives wins because N is so much bigger than Q. I don't do the calculation that way. I'm closer to equal weight, right.
Eliezer: Yeah, you're going to have trouble putting that on a firm epistemic foundation but Nick Bostrom has done some work on what he calls parliamentary models of decision-making. I'm not sure Nick Bostrom would endorse their extension to this case, but descriptively, it seems a lot of what we do is sort of like the different things we think might be true get to be voices in our head in proportion to how true they are and then they negotiate with each other. This has the advantage of being robust against Pascal's Mugging-type stuff, which I'd like to once again state for the historical record: I invented that term and not as something that you ought to do! So anyway, it's robust against Pascal's Mugging-type stuff, and it has the disadvantage of plausibly failing the what-if-everyone-did-that test.
***
Holden: Let me step back a second. I hear your claim that I should assign a very high probability that we can — if we survive — colonize the stars. I believe this to be something that smart technical people would not agree with. I've outlined why I think they wouldn't agree with it, but not done a great job with it and that's something that I'd be happy to think more about and talk more about.
Eliezer: Are there reasons apart from the Fermi Paradox?
Holden: I don't know what all the reasons are. I've given my loose impression and it's not something that I've looked into much, because I didn't really think there was anyone on the other side.
The post Existential Risk Strategy Conversation with Holden Karnofsky appeared first on Machine Intelligence Research Institute.


2013 in Review: Outreach

This is the 2nd part of my personal and qualitative self-review of MIRI in 2013.
By "outreach" I refer to general outreach efforts, rather than e.g. outreach to specific researchers, which will be discussed in the post about MIRI's 2013 research activities.

Outeach in 2013

In early 2013, we decided to reduce our outreach efforts significantly, and we did.
However, we learned throughout 2013 that some forms of "indirect" outreach tend to be pretty cost-effective for MIRI's goals.1
Therefore, we plan to put more effort into indirect outreach in 2014 than we did in 2013.

Reducing our outreach efforts
MIRI provisionally decided to reduce its outreach efforts in mid-2012, when we decided to enter negotiations with Singularity University (SU) in response to their offer to acquire the Singularity Summit, a deal finally closed in December 2012. The reduction in outreach efforts explained in our 2013 strategy post was largely a continuation of this earlier strategic decision, which had been conditional on the deal with SU closing successfully. (We would have continued to run the Singularity Summit had SU not acquired it.)
MIRI will co-produce the Singularity Summit for the next couple years, but this requires minimal effort on our part, and the scheduling of future Summits is in SU's hands. SU has not yet scheduled the next Singularity Summit.
Without a Singularity Summit occurring in 2013, our direct outreach efforts this year were few. By "direct outreach efforts" I refer to efforts that attempt to directly expose people to MIRI's mission focused on the superintelligence control problem. This is in contrast to "indirect outreach efforts," which aim to expose people not to MIRI's mission, but to other ideas that may begin to bridge the inferential distance that lies between common knowledge and the core ideas motivating MIRI's mission — ideas such as effective altruism, applied rationality, global catastrophic risks, the security mindset, etc.
Our direct outreach efforts in 2013 were:

A new website, a necessary part of our rebranding as the Machine Intelligence Research Institute, and some increased use of social media.
A few general-audience talks about superintelligence control, given at universities and technology companies (Stanford, Oxford, Facebook, Quixey, and Heroku), and also at the Effective Altruism Summit.
Two Bay Area events we hosted for our community: our April "relaunch as MIRI" event, and our September book launch event for James Barrat's Our Final Invention. Two MIRI staffers also published reviews of Barrat's book, here and here.
Two new ebooks containing previously-written content: Facing the Intelligence Explosion and The Hanson-Yudkowsky AI-Foom Debate.
We also attended several events for networking purposes, and had conversations with interested parties who contacted us, and thereby spoke to many people about superintelligence control.
Various small writings and interviews, e.g. a short article for Quartz and an interview for Big Picture Science.

It is difficult to measure how much value these efforts are producing for our long-term goals. Some basic statistics on these efforts are recorded in a footnote.2
Overall, my own impressions on the value of these direct research efforts are as follows:

The new domain name (intelligence.org) and website seem clearly worthwhile. Our limited use of social media also seems worthwhile, not to grow our community (because: too much inferential distance), but to keep our existing community engaged with MIRI's work. (For many people, our Facebook updates are the easiest way to stay in touch with MIRI.)
Our superintelligence control talks at the Effective Altruism Summit were worthwhile because there was less inferential distance with that audience. We also know that some donations came as a result of those talks. In contrast, we haven't yet received any specific evidence that the other superintelligence control talks produced value toward our mission, other than by keeping us in touch with our existing community. We didn't strongly expect to see such evidence so soon, but this still constitutes non-negligible evidence of absence of positive outreach effects.
The two Bay Area events were valuable for staying in touch with our existing community, but they failed to produce major press stories even though we invited several major journalists to each. The events cost more staff time and attention than money. Our two book reviews of Our Final Invention are too recent for us to know whether they caused useful action (beyond merely purchasing the book) in their readers.
The two ebooks of pre-existing content were inexpensive to produce, and they make our material more accessible. They are too recent for us to have specific evidence of positive effects, but I'd guess they were worth the expense.
As far as I can tell, the networking events at which we spoke to people about superintelligence control (rather than effective altruism, rationality, etc.) seem to have produced negligible value. We experimented with several presentations of our material, but I think the inferential distance is just too great to engage new people who aren't already familiar with effective altruism, applied rationality, global catastrophic risks, etc.

Indirect outreach efforts were more clearly successful
Our indirect outreach efforts in 2013 were also aimed at long-term effects:

Eliezer Yudkowsky wrote new chapters of Harry Potter and the Methods of Rationality (HPMoR), and MIRI distributed 2500+ paperback copies of its first 17 chapters to schools, universities, and companies.3

Carl Shulman contributed research and career counseling services to the Center for Effective Altruism (CEA).
We produced several front-page posts for the Less Wrong community, e.g. a Decision Theory F.A.Q.4

The positive effects of our posts to Less Wrong are hard to measure, but our other indirect outreach efforts have produced clear evidence of their value more quickly than our direct outreach efforts have.
Carl's work with CEA had measurable effects on the funding and attention going into long-term philanthropy (including the superintelligence control problem).
Finally, some of the evidence we've gotten about the indirect outreach value of HPMoR:

When I observe someone  doing something useful to MIRI's mission (e.g. donation or research) for the first time, I typically ask how they came to be involved, and I trace the story back to their first contact with someone at MIRI, with one of MIRI's close friends, or to something that MIRI or MIRI's close friends produced. HPMoR is now the single most common "first contact" I encounter. (The 2nd most common first contact is with The Sequences.)
The paperback version of HPMoR ends in the middle of chapter 17, meaning that readers coming to HPMoR.com after finishing the paperback should jump directly to chapter 17. According to Google Analytics, people visit chapter 17 significantly more often than any other chapter outside the first 10 chapters and newly released chapters. It's hard to tell exactly, but my best guess is that 100-400 people per month are coming to hpmor.com as a result of first reading the paperback copy of the first 17 chapters.

These data fit with MIRI's earlier experience of the effects of its direct and indirect outreach efforts. For example, as I said earlier: Eliezer Yudkowsky spent MIRI's early years appealing directly to people about the importance of superintelligence control research. Some helpful people found his work, but the audience was being filtered for "interest in future technology" rather than "able to reason well," and thus when Eliezer would make basic arguments for e.g. the orthogonality thesis or convergent instrumental values, the responses he would get were typically akin to the output of a free association task. So Eliezer wrote The Sequences and HPMoR, for which the primary audience filter is "interest in improving one's reasoning." This audience, in our experience, is more likely to do useful things when we present the case for effective altruism, for x-risk reduction, for FAI research, etc.

Lessons for 2014
Today I find myself even more confident than I was in April 2013 that certain kinds of indirect outreach by MIRI are more likely to produce value for superintelligence control than MIRI's direct outreach efforts are. For this reason, I suspect that in 2014 we will continue to scale back our direct outreach efforts.5 Meanwhile, we are likely to increase our indirect outreach efforts, in particular by giving Eliezer time to write more HPMoR chapters, by distributing more HPMoR paperbacks, and by publishing an ebook of The Sequences.
We will also likely provide some consulting and other support for organizations who are doing the kinds of outreach that end up benefiting MIRI (and other organizations), for example CFAR (rationality) and CEA (effective altruism).
We are currently gathering much more information about the effects of our direct and indirect outreach efforts, but these data will take several months to gather. It's possible those data will overturn the basic conclusions in this post, but it's more likely they will slightly adjust them.Basic statistics on direct outreach efforts: (1) Our new intelligence.org domain name rose from PageRank 0 to PageRank 5, one rank below LessWrong.com, a site that has been heavily linked and active since 2009. (2) Since it launched in March 2013, intelligence.org has averaged 28,613 unique visitors per month. This is roughly double the traffic to our previous domain, singularity.org, which averaged 14,843 unique visitors per month during its operation (July 2012 through February 2013). (3) Our other "mini-sites" got negligible traffic, as expected. During their active months in 2013, here's how many unique visitors per month our mini-sites got: IntelligenceExplosion.com (2,336/mo), Friendly-AI.com (50/mo), TheUncertainFuture.com (129/mo). (4) As of Jan. 2nd, MIRI's Facebook page had ~5000 Likes, had been seen by ~19,500 unique people, and had "engaged" ~1100 unique people (meaning that they liked, commented on, or clicked on MIRI's Facebook wall posts). MIRI's Google+ page is more recent and was much less active: with ~230 +1s (analogous to Facebook Likes), and 17 re-shares. Note that our Facebook Likes count appears to be artificially high. From Oct. 9th to Nov. 9th, MIRI's Facebook page received an unusual number of Likes. This began again on Dec. 22nd. We don't know the cause. (5) Our newsletter subscription count grew only slightly, from 9,152 subscribers in January 2013 to 10,024 subscribers as of December 22nd. (6) Our April 2013 relaunch event was attended by ~65 people. The September 2013 book launch event was attended by ~75 people. In both cases, most attendees were close friends of MIRI. (7) Within one month of its publication, Luke's review of Our Final Invention on KurzweilAI.net saw ~8,500 unique pageviews and was responsible for at least 20 purchases of the book, which describes MIRI's mission and work in some detail. Meanwhile, Louie's review of Our Final Invention for Singularity Hub was responsible for at least 60 purchases of the book as of Jan. 2nd, 2014. (8) From April-Nov 2013, Facing the Intelligence Explosion was purchased 1,379 times. Since September, The Hanson-Yudkowsky AI-Foom Debate has been downloaded at least 1,240 times, but this counts only downloads from people who clicked a link to the book from intelligence.org. (9) How many people attended our general-audience talks? By memory, I'd guess 40 at Stanford, 30 at Oxford, 20 at Facebook, 40 at Quixey, 15 at Heroku, and 50 at the Effective Altruism Summit. These were smaller, more targeted talks than e.g. our past talks at the Singularity Summits.HPMoR is Yudkowsky's private project, but MIRI sponsored this work in the sense that Yudkowsky remained on salary at MIRI while he took time off from "direct MIRI work" to write and release new chapters of HPMoR.MIRI's substantive, outreach-oriented, front-page posts for the Less Wrong community in 2013 were: (1) Decision Theory F.A.Q.; (2) Fermi Estimates; (3) Start Under the Streetlight, then Push into the Shadows; (4) Four Focus Areas of Effective Altruism; (5) How to Measure Anything; (6) MetaMed: Evidence-Based Healthcare.There will probably be at least two major exceptions to this trend. First, in early 2014 we will publish an ebook called Smarter Than Us, which was written by Stuart Armstrong all the way back in late 2012. We just didn't have staff time to turn it into an ebook until very recently. Second, we will probably publish something about the upcoming film Transcendence, because that is a unique opportunity to direct many eyeballs to our writings. Also, we will likely assist in the promotion of Nick Bostrom's Superintelligence book, but this really qualifies as "outreach to researchers" rather than "general outreach" because Bostrom's book is a scholarly monograph.The post 2013 in Review: Outreach appeared first on Machine Intelligence Research Institute.

2013 in Review: Outreach


Want to help MIRI by investing in XRP?

Recently, Mt.Gox and Ripple creator Jed McCaleb gave MIRI a large donation in XRP, which is the #2 cryptocurrency in market cap behind Bitcoin. This gift, along with our recent successful fundraiser, should enable us to hire the beginnings of a full-time Friendly AI research team in 2014. It's difficult to know exactly how to value the donation, but (e.g.) valuing it at the exchange rate at the time of donation would make it the largest single donation to MIRI in our history.
The size of Jed's donation means that we currently have more XRP than it makes sense to hold in our diversified asset portfolio. To reduce our XRP holdings while growing the Ripple user base, we'd like to sell some of our XRP to members of our community. If you're interested, please contact Malo Bourgon (<EMAIL>).
How does Ripple/XRP work? Technically, Ripple is an online payments protocol and XRP is an associated digital currency, but XRP is often referred to as "Ripple," which can be confusing. The best explanation of all this yet written is "Bitcoin Vs. Ripple" from the Coinsetter blog, which I'll excerpt below:
…think of Ripple as "Kayak for currency exchange." Ripple will compare various pathways of exchanging one currency to another and find the lowest cost option. Transactions also happen rapidly (in under 5 seconds)…
XRP is a digital currency… with an embedded use, which is to pay for Ripple transactions, create Ripple accounts, and be a currency of last resort in the Ripple system… Whereas Bitcoin has no inherent use (its demand and use is solely based on people's interest in Bitcoin), XRP has… a basic reason why someone may need to use XRP, which is to complete Ripple transactions.
…Should Bitcoin believers be worried that Ripple is on its way to crush their dream currency? Absolutely not. Interest in Bitcoin will continue to grow, and separately, interest in Ripple will continue to grow. In fact, the Ripple network should help make bitcoins substantially easier to buy and sell, as well as legitimize them.
Click here to create a Ripple wallet and join the network; the process takes less than 60 seconds.
MIRI continues to accept donations in both XRP and Bitcoin, and we send tax receipts for non-anonymous donations in either currency.
The post Want to help MIRI by investing in XRP? appeared first on Machine Intelligence Research Institute.

Want to help MIRI by investing in XRP?


MIRI's January 2014 Newsletter

Dear friends,Wow! MIRI's donors finished our Winter 2013 fundraising drive three weeks early. Our sincere thanks to everyone who contributed.
Research Updates

MIRI's December workshop resulted in 7 new technical reports and one new paper. Photos from the workshop are here.
New paper by Luke Muehlhauser and Nick Bostrom: "Why We Need Friendly AI." io9's George Dvorsky interviewed Luke about the paper here.
Eliezer Yudkowsky and Robby Bensinger have begun to describe another open problem in Friendly AI: naturalized induction.
Two new interviews: Josef Urban on machine learning and automated reasoning, and DARPA's Kathleen Fisher on high-assurance systems.

Other Updates

We published the first part of our 2013 in Review series, on operations.
We published a transcript and "summary of disagreements" for a conversation about MIRI strategy held with Jacob Steinhardt (Stanford), Holden Karnofsky (GiveWell), and Dario Amodei (Stanford).
We published our first donor story: "Noticing Inferential Distance."

Other Links

FQXI is holding an essay contest on the subject of "How Should Humanity Steer the Future?" Submission deadline is April 18th. First prize is $10,000, and up to 20 entries may win $1,000 or more.
MIT's Max Tegmark on AI x-risk at the HuffPost blog.
"A Long-run Perspective on Strategic Cause Selection and Philanthropy," by Nick Beckstead and Carl Shulman.
In case you missed it last year, Bostrom's "Existential Risk Prevention as Global Priority" remains the best extant summary of existential risks in general.

As always, please don't hesitate to let us know if you have any questions or comments.
Best,

Luke Muehlhauser
Executive Director

The post MIRI's January 2014 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s January 2014 Newsletter


MIRI strategy conversation with Steinhardt, Karnofsky, and Amodei

On October 27th, 2013, MIRI met with three additional members of the effective altruism community to discuss MIRI's organizational strategy. The participants were:

Eliezer Yudkowsky (research fellow at MIRI)
Luke Muehlhauser (executive director at MIRI)
Holden Karnofsky (co-CEO at GiveWell)
Jacob Steinhardt (grad student in computer science at Stanford)
Dario Amodei (post-doc in biophysics at Stanford)

We recorded and transcribed much of the conversation, and then edited and paraphrased the transcript for clarity, conciseness, and to protect the privacy of some content. The resulting edited transcript is available in full here (62 pages).
Our conversation located some disagreements between the participants; these disagreements are summarized below. This summary is not meant to present arguments with all their force, but rather to serve as a guide to the reader for locating more information about these disagreements. For each point, a page number has been provided for the approximate start of that topic of discussion in the transcript, along with a phrase that can be searched for in the text. In all cases, the participants would likely have quite a bit more to say on the topic if engaged in a discussion on that specific point.

Summary of disagreements
Page 7, starting at "the difficulty is with context changes":

Jacob: Statistical approaches can be very robust and need not rely on strong assumptions, and logical approaches are unlikely to scale up to human-level AI.
Eliezer: FAI will have to rely on lawful probabilistic reasoning combined with a transparent utility function, rather than our observing that previously executed behaviors seemed 'nice' and trying to apply statistical guarantees directly to that series of surface observations.

Page 10, starting at "a nice concrete example"

Eliezer: Consider an AI that optimizes for the number of smiling faces rather than for human happiness, and thus tiles the universe with smiling faces. This example illustrates a class of failure modes that are worrying.
Jacob & Dario: This class of failure modes seems implausible to us.

Page 14, starting at "I think that as people want":

Jacob: There isn't a big difference between learning utility functions from a parameterized family vs. arbitrary utility functions.
Eliezer: Unless 'parameterized' is Turing complete it would be extremely hard to write down a set of parameters such that human 'right thing to do' or CEV or even human selfish desires were within the hypothesis space.

Page 16, starting at "Sure, but some concepts are":

Jacob, Holden, & Dario: "Is Terry Schiavo a person" is a natural category.
Eliezer: "Is Terry Schiavo a person" is not a natural category.

Page 21, starting at "I would go between the two":

Holden: Many of the most challenging problems relevant to FAI, if in fact they turn out to be relevant, will be best solved at a later stage of technological development, when we have more advanced "tool-style" AI (possibly including AGI) in order to assist us with addressing these problems.
Eliezer: Development may be faster and harder-to-control than we would like; by the time our tools are much better we might not have the time or ability to make progress before UFAI is an issue; and it's not clear that we'll be able to develop AIs that are extremely helpful for these problems while also being safe.

Page 24, starting at "I think the difference in your mental models":

Jacob & Dario: An "oracle-like" question-answering system is relatively plausible.
Eliezer: An "oracle-like" question-answering system is really hard.

Page 24, starting at "I don't know how to build":

Jacob: Pre-human-level AIs will not have a huge impact on the development of subsequent AIs.
Eliezer: Building a very powerful AGI involves the AI carrying out goal-directed (consequentialist) internal optimization on itself.

Page 27, starting at "The Oracle AI makes a":

Jacob & Dario: It should not be too hard to examine the internal state of an oracle AI.
Eliezer: While AI progress can be either pragmatically or theoretically driven, internal state of the program is often opaque to humans at first and rendered partially transparent only later.

Page 38, starting at "And do you believe that within having":

Eliezer: I've observed that novices who try to develop FAI concepts don't seem to be self-critical at all or ask themselves what could go wrong with their bright ideas.
Jacob & Holden: This is irrelevant to the question of whether academics are well-equipped to work on FAI, both because this is not the case in more well-developed fields of research, and because attacking one's own ideas is not necessarily an integral part of the research process compared to other important skills.

Page 40, starting at "That might be true, but something":

Holden: The major FAI-related characteristic that academics lack is cause neutrality. If we can get academics to work on FAI despite this, then we will have many good FAI researchers.
Eliezer: Many different things are going wrong in the individuals and in academia which add up to a near-total absence of attempted — let alone successful — FAI research.

Page 53, starting at "I think the best path is to try":

Holden & Dario: It's relatively easy to get people to rally (with useful action) behind safety issues.
Eliezer: No, it is hard.

Page 56, starting at "My response would be that's the wrong thing":

Jacob & Dario: How should we present problems to academics? An English-language description is sufficient; academics are trained to formalize problems once they understand them.
Eliezer: I treasure such miracles when somebody shows up who can perform them, but I don't intend to rely on it and certainly don't think it's the default case for academia. Hence I think in terms of MIRI needing to crispify problems to the point of being 80% or 50% solved before they can really be farmed out anywhere.

This summary was produced by the following process: Jacob attempted a summary, and Eliezer felt that his viewpoint was poorly expressed on several points and wrote back with his proposed versions. Rather than try to find a summary both sides would be happy with, Jacob stuck with his original statements and included Eliezer's responses mostly as-is, and Eliezer later edited them for clarity and conciseness. A Google Doc of the summary was then produced by Luke and shared with all participants, with Luke bringing up several points for clarification with each of the other participants. A couple points in the summary were also removed because it was difficult to find consensus about their phrasing. The summary was published once all participants were happy with the Google Doc.
The post MIRI strategy conversation with Steinhardt, Karnofsky, and Amodei appeared first on Machine Intelligence Research Institute.


Kathleen Fisher on High-Assurance Systems

Dr. Kathleen Fisher joined DARPA as a program manager in 2011. Her research and development interests relate to programming languages and high assurance systems. Dr. Fisher joined DARPA from Tufts University. Previously, she worked as a Principal Member of the technical staff at AT&T Labs. Dr. Fisher received her Doctor of Philosophy in computer science and her Bachelor of Science in math and computational science from Stanford University.

Luke Muehlhauser: Kathleen, you're the program manager at DARPA for HACMS program, which aims to construct cyber-physical systems which satisfy "appropriate safety and security properties" using a "clean-slate, formal methods-based approach." My first question is similar to one I asked Greg Morrisett about the SAFE program, which aims for a "clean slate design for resilient and secure systems": In the case of HACMS, why was it so important to take the "clean slate" approach, and design the system from the ground up for safety and security (along with functional correctness)?

Kathleen Fisher: Researchers have been trying to prove programs correct for decades, with very little success until recently (successful examples include NICTA's seL4 microKernel and INRIA's compCert verifying C compiler). One of the lessons learned in this process is that verifying existing code is much harder than verifying code written with verification in mind.
There are a couple of reasons for this. First, many of the invariants that have to hold for a program to be correct often exist only in the head of the programmer. When trying to verify a program after the fact, these invariants have been lost and can take a very long time for the verifier to rediscover. Second, sometimes the code can be written in multiple ways, some of which are much easier to verify than others. If programmers know they have to verify the code, they'll choose the way that is easy to verify.

Luke: Harper (2000) quotes an old message by Ken Frith from the Safety Critical Mailing List:
The thought of having to apply formal proofs to intelligent systems leaves me cold. How do you provide satisfactory assurance for something that has the ability to change itself during a continuous learning process?
Obviously we can't write a formal specification of the entire universe and formally verify a system against it, so what kinds of (formal) things can we do to ensure desirable behavior from increasingly capable autonomous systems?

Kathleen: Verifying adaptive or intelligent systems is a difficult challenge. It is the subject of active research, particularly in the avionics industry. One approach is to use a Simplex Architecture, in which the intelligent system is monitored to ensure it stays within a safety envelop. If the intelligent system looks likely to leave the envelop, a monitor diverts control to a basic system. In this approach, the monitor, the switching code, and the basic system have to be verified directly, but not the intelligent system. Verifying these smaller pieces is within the reach of current technology.

Luke: You said earlier that "verifying existing code is much harder than verifying code written with verification in mind." This suggests that when a very high level of system safety is needed, e.g. for autopilot software, a great deal of extra work is required, because one must essentially start from scratch to design a system "from the ground up" with safety and verification in mind.
Moreover, I have the impression that the work required to design a verifiably safe intelligent system tends to be far more "serial" in nature than the work required to design an intelligent system without such a high degree of verifiable safety. When designing an intelligent system that doesn't need to be particularly "safe", e.g. a system for intelligently choosing which ads to show website visitors, one can cobble together bits of code from here and there and test the result on some data sets and then run the system "in the real world." But a verifiably safe system needs to be built in a very precise way, with many algorithm design choices depending on previous choices, and with many research insights dependent on the shape of previous insights, etc. In this way, it seems that the project of designing a not-verifiably-safe intelligent system parallelizes more easily than the project of designing a verifiably safe intelligent system.
Is this also your impression? How would you modify my account?

Kathleen: Your assessment is accurate. Basically, if your code has a wide range of acceptable behaviors and if there aren't particularly bad consequences if it misbehaves, it is a lot easier to write code.
I'd be careful with the term parallelize. It is usually used to refer to programs where parts run at the same time as other parts, say on multiple computers or chips. It can also refer to programmers themselves working in parallel. The way you are using it, it reads like you mean the code is running in parallel when I think you mean to say the code is written in parallel. Both are easier for non-high-assurance code, but the argument you are making lends itself to the "written in parallel" interpretation.

Luke: What qualitative or quantitative trends do you observe or anticipate with regard to verifiably safe cyber-physical systems? For example: Do you think research on the capability and intelligence of cyber-physical systems is outpacing research on their safety and verifiability? How much funding and quality-adjusted researcher hours are going into one (capability and intelligence) relative to the other (safety and verifiability), with respect to cyber-physical systems specifically? How much of the low-hanging fruit seems to have been plucked with respect to each, such that additional units of progress require more funding and researcher time than previous units?

Kathleen: In general, research into capabilities outpaces the corresponding research into how to make those capabilities secure. The question of security for a given capability isn't interesting until that capability has been shown to be possible, so initially researchers and inventors are naturally more focused on the new capability rather than on its associated security. Consequently, security often has to catch up once a new capability has been invented and shown to be useful.
In addition, by definition, new capabilities add interesting and useful new capabilities, which often increase productivity, quality of life, or profits. Security adds nothing beyond ensuring something works the way it is supposed to, so it is a cost center rather than a profit center, which tends to suppress investment.
I don't know how one would go about collecting the data to show levels of investment in research for capabilities vs. security. I'm not sure the dollar totals would be that revealing. Research in security could still be lagging research in capabilities even if more money were being spent on security, if research in security were more costly than research into capabilities.

Luke: It is one challenge to ensure the safe behavior of a system with very limited behaviors and very limited decision-making capacity, and quite another to ensure the safe behavior of a system with much greater autonomous capability. What do you think are the prospects for ensuring the safe behavior of cyber-physical systems as they are endowed with sufficient intelligence and autonomy to e.g. replace factory workers, replace taxi drivers, replace battlefield troops, etc. over the next several decades?

Kathleen: As you say, it is much easier to ensure the safe and secure behavior of simple systems with well-defined behavior. That said, it is clear that we will be fielding more and more complex cyber-physical systems for all kinds of activities and we will need to be able to ensure that those systems are sufficiently safe and secure. I think a combination of techniques will enable us to reach this goal, although substantial additional research is needed. These techniques include: 1) model-based design, 2) program synthesis, 3) security- and safety-aware composition, and 4) simplex-based architectures. Basically, we'll need to start with simple components and simple architectures, and then leverage them to build more and more complex systems.

Luke: Thanks!
The post Kathleen Fisher on High-Assurance Systems appeared first on Machine Intelligence Research Institute.


Donor Story #1: Noticing Inferential Distance

2013 was by far MIRI's most successful fundraising year (more details later), so now we're talking to our donors to figure out: "Okay, what exactly are we doing so right?"
Below is one donor's story, anonymized and published with permission:
My decision to donate was heavily dependent upon MIRI's relationship with LessWrong. I did look into MIRI itself, perused the blog and the papers, and did some fact-checking. But this was largely sanity-checking after I had been convinced to donate by my interactions on LessWrong.
Initially, I wasn't so much convinced to donate as I was convinced that FAI is a more pressing problem than my prior concerns. Once I believed this, it wasn't a question of whether I was going to donate to FAI research but a question of where to focus my efforts.
I chose to donate to MIRI after it became apparent to me that

Few people are aware of the problems posed by uFAI.
Among those who are, many dismiss the problem after failing to understand it.

This perhaps sounds arrogant. Allow me to explain some:
Perhaps paradoxically, one of the biggest factors that made me trust MIRI was Holden Karnofsky's critique of the organization. Holden's tool AI suggestion and a number of his other objections seemed, to my eye, transparently foolish. Afterwards, I read the responses of Luke and Eliezer, who rejected these points for reasons similar to why I had rejected them. This went a long way towards convincing me that whatever organization Luke and Eliezer were working at was particularly important.
This scenario played out many times over, in big discussions and in small comments. I discovered that there were many people objecting to uFAI as a problem, and that the vast majority of them fundamentally misunderstood the concerns.
It's not that I think Holden Karnofsky or other dissenters are stupid, nor misinformed. I respect Karnofsky in particular: he's very intelligent and he's doing incredible work. I certainly don't mean to imply I'm smarter than him nor any others who question MIRI's mission.
Rather, I have a history of arguing for uncouth ideas. I've often found myself on the far side of an inferential gap. I've long been convinced that smart people can fundamentally misunderstand the most important problems. Frequently I have tried to hone a strange idea, only to have discussion partners fall into the inferential gap four steps before we got the real questions.
Many of the discussions about FAI — Karnofsky's critique primary among them — convinced me that FAI is the same sort of problem. The sort where people get caught up debating whether the problem is actually a problem, the sort where it's very difficult to find people who are actually searching for solutions instead of debating whether solutions are necessary.
This was my primary reason for trusting MIRI: these discussions left me convinced that I should expect a dearth of experts who are appropriately concerned.
I didn't expect there to exist organizations outside of MIRI (and FHI, and a few others in the same circles) that I could trust to address the problems as I see them. The number of smart people stumbling on inferential gaps made me cautious of other organizations sharing similar missions: even if their goals sounded good it would have been difficult to convince me that their leadership could avoid the common pitfalls.
By contrast, Luke, Eliezer, and Nick Bostrom demonstrated their abilities time and again, across many blog posts, discussions, and papers. They slowly built up the trust that I now place in MIRI and FHI.
I understand that this still sounds arrogant, and this was not lost on me. I was introduced to these issues via Eliezer's writings, and thus I had to expect a bias towards Eliezer's viewpoint. The fact that I was dismissing many arguments from very smart people with minimal consideration raised some red flags. The fact that the field was rife with misconception implied that I should anticipate misconceptions on my own part.
I was aware of all this, and I spent a long time trying to account for these points. The decision was not made lightly: it took four solid months of reflection, research, and thought before I was confident enough to donate to MIRI. In that timeframe I found more respect for the opposing views, but my conclusions did not change.
I came out the other end convinced that uFAI is a pressing problem, that FAI research is important, and that the field is full of landmines. MIRI remains one of very few organizations that has convinced me they have a chance of navigating this minefield successfully.
Of course, these conclusions were not quite so crisp in my head, at the time. There was always more thinking to do, more reflection to be had, more facts to check before moving piles of money around. And life kept on happening in the meantime: work and leisure took precedent over deciding where my money should go. It was the summer matching challenge that finally forced my hand, shook me from my reverie, and provided a convenient excuse to actually hit the button.
The post Donor Story #1: Noticing Inferential Distance appeared first on Machine Intelligence Research Institute.


7 New Technical Reports, and a New Paper

Recently, MIRI released 7 brief technical reports that explain several pieces of theoretical progress made at our December 2013 research workshop. Several of these results build on work done at our July and November 2013 workshops, and also on between-workshop research by Paul Christiano, Benja Fallenstein, and others.
To understand these technical reports in context, and to discuss them, please see Benja Fallenstein's post: Results from MIRI's December workshop. See also two posts about the workshop by workshop participant John Baez.
The 7 technical reports are:

Hahn, "Scientific Induction in Probabilistic Metamathematics"
Yudkowsky, "The Procrastination Paradox"
Fallenstein, "An infinitely descending sequence of sound theories each proving the next consistent"
Soares, "Fallenstein's Monster"
Stiennon, "Recursively-defined logical theories are well-defined"
Fallenstein, "The 5-and-10 problem and the tiling agents formalism"
Fallenstein, "Decreasing mathematical strength in one formalization of parametric polymorphism"

Also, Nik Weaver attended the first day of the workshop, and gave a tutorial on his response paper to Yudkowsky & Herreshoff's tiling agents paper, titled "Paradoxes of rational agency and formal systems that verify their own soundness." Benja Fallenstein's comments on Weaver's idea of "naturalistic trust" are here: Naturalistic trust among AIs.
The post 7 New Technical Reports, and a New Paper appeared first on Machine Intelligence Research Institute.


Winter 2013 Fundraiser Completed!

Wow! MIRI's donors just finished our 2013 winter matching challenge three weeks ahead of the deadline!
As it happens, the person who finished this drive was Patrick LaVictoire, who says his donation was the product of his altruism tip jar habit.
I did not expect to finish the fundraiser so quickly. Basically, this happened because the 3:1 matching for "new large donors" worked even better than we hoped: this fundraiser brought forth ten "new large donors," an uncommonly high number.
For those who were wondering whether they should contribute to MIRI or to its child organization CFAR, the answer is now clear: please contribute to CFAR's ongoing fundraiser.
Many, many thanks to everyone who contributed!
The post Winter 2013 Fundraiser Completed! appeared first on Machine Intelligence Research Institute.

Winter 2013 Fundraiser Completed!