Laurent Orseau on Artificial General Intelligence

 Laurent Orseau is an associate professor (maître de conférences) since 2007 at AgroParisTech, Paris, France. In 2003, he graduated from a professional master in computer science at the National Institute of Applied Sciences in Rennes and from a research master in artificial intelligence at University of Rennes 1. He obtained his PhD in 2007. His goal is to build a practical theory of artificial general intelligence. With his co-author Mark Ring, they have been awarded the Solomonoff AGI Theory Prize at AGI'2011 and the Kurzweil Award for Best Idea at AGI'2012.

Luke Muehlhauser: In the past few years you've written some interesting papers, often in collaboration with Mark Ring, that use AIXI-like models to analyze some interesting features of different kinds of advanced theoretical agents. For example in Ring & Orseau (2011), you showed that some kinds of advanced agents will maximize their rewards by taking direct control of their input stimuli — kind of like the rats who "wirehead" when scientists give them direct control of the input stimuli to their reward circuitry (Olds & Milner 1954). At the same time, you showed that at least one kind of agent, the "knowledge-based" agent, does not wirehead. Could you try to give us an intuitive sense of why some agents would wirehead, while the knowledge-based agent would not?

Laurent Orseau: You're starting with a very interesting question!
This is because knowledge-seeking has a fundamental distinctive property: On the contrary to rewards, knowledge cannot be faked by manipulating the environment. The agent cannot itself introduce new knowledge in the environment because, well, it already knows what it would introduce, so it's not new knowledge. Rewards, on the contrary, can easily be faked.
I'm not 100% sure, but it seems to me that knowledge seeking may be the only non-trivial utility function that has this non-falsifiability property. In Reinforcement Learning, there is an omnipresent problem called the exploration/exploitation dilemma: The agent must both exploit its knowledge of the environment to gather rewards, and explore its environment to learn if there are better rewards than the ones it already knows about. This implies in general that the agent cannot collect as many rewards as it would like.
But for knowledge seeking, the goal of the agent is to explore, i.e., exploration is exploitation. Therefore the above dilemma collapses to doing only exploration, which is the only meaningful unified solution to this dilemma (the exploitation-only solution leads either to very low rewards or is possible only when the agent already has knowledge of its environment, as in dynamic programming). In more philosophical words, this unifies epistemic rationality and instrumental rationality.
Note that the agent introduced in Orseau & Ring (2011), and better developed in Orseau (2011) where a convergence proof is given, works actually only for deterministic environments. Its problem is that it may consider noise as information, and get addicted to it, i.e., it may stare at a detuned TV screen forever. And one could well consider this as "self-delusion".
Fortunately, with Tor Lattimore and Marcus Hutter, we are finalizing a paper for ALT 2013 where we considered all computable stochastic environments. This new agent does not have the defective behavior of the 2011 agent, and I think it would even have a better behavior even in deterministic environments. For example, it would (it seems) not focus for too long on the same source of information, and may from time to time get back to explore the rest of the environment before eventually coming back to the original source; i.e., it is not a monomaniac agent.
A side note: If you (wrongly) understand knowledge-seeking as learning to predict all possible futures, then a kind of self-delusion may be possible: The agent might just jump into a trap, where all its observations would be the same whatever its actions, and it would thus have converged to optimal prediction. But we showed that the knowledge-seeking agent would give no value to such actions.

Luke: In two other papers, you and Mark Ring addressed a long-standing issue in AI: the naive Cartesian dualism of the ubiquitous agent-environment framework. Could you explain why the agent-environment framework is Cartesian, and also what work you did in those two papers?

Laurent: In the traditional agent framework, we consider that the agent interacts with its environment by sending, at each interaction cycle, an action that the environment can take into account to produce an observation, that the agent can in turn take into account to start the next interaction cycle and output a new action. This framework is very useful in practice because it avoids a number of complications of the real life. Those complications are exactly what we wanted to address head on. Because at some point you need to pull your head out of the sand and start dealing with the complex but important issues. But certainly many people, and in particular people working with robots, are quite aware that the real world is not a dualist framework. So in a sense, it was an obvious thing to do, especially because it seems that no-one had done it before, at least from this point of view and to the best of our knowledge.
The traditional framework is dualist in the sense that it considers that the "mind" of the agent (the process with which the agent chooses its actions) lies outside of the environment. But we all know that if we ever program an intelligent agent on a computer, this program and process will not be outside of the world, they will be a part of it and, even more importantly, computed by it. This led us to define our space-time embedded intelligence framework and equation.
Put simply, the idea is to consider an (realistic) environment, and a memory block of some length on some computer or robot in this environment. Then what is the best initial configuration of the bits on this memory block according to some measure of utility upon the expected future history?
Some people worry that this is too general (in particular if you just consider some block of bits in an environment, not necessarily on a computer) and that we lose the essence of agency, which is to deal with inputs and outputs. But they forget that a) this systemic framework does also allow for defining stones (simply ignore the inputs and output a constant value) and b) this is how the real world is: If an AGI can duplicate itself and split itself in so many parts on many computers, robots and machines, how can we really identify this agent as a single systemic entity?
Some other people worry about how this framework could be used in practice, and that it is too difficult to deal with. Our goal was not to define a framework where theorems are simple to prove and algorithms simple to write, but to define a framework for AGI that is closer to the real world. If the latter is difficult to deal with, then so be it. But don't blame the framework, blame the real world. Anyway, we believe there are still interesting things to do with this framework. And I believe it is at least still useful to help people not forget that the real world is different from the usual text book simplifications. This is probably not very relevant for applied machine learning and narrow AI research, but I believe it is very important for AGI research.
However, let me get one thing straight: Even in the traditional framework, an agent can still predict that it may be "killed" (in some sense), for example if an anvil falls on its body. This is possible if the body of the agent, excluding the brain but including its sensors and effectors, are considered to be part of the environment: The agent can then predict that the anvil will destroy them and it will be unable to get any information and reward from and perform any action to the environment. Whenever we consider our skull (or rather, the skull of the robot) and brain to always be unbreakable, non-aging, and not subject to drugs, alcohol and external events such as heat, accelerations and magnetic waves, we can quite safely use the traditional framework.
But remove one of these hypotheses and the way the agent computes may become different from what it assumes it would be, hence leading to different action selection schemes. Regarding artificial agents, tampering with a source code is even easier than with a human brain. AGIs of the future will probably face a gigantic number of cracking and modification attempts, and the agent itself and its designers should be well aware that this source code and memory are not in a safe dualist space. In the "Memory issues of intelligent agents" paper, we considered various consequences of giving the environment the possibility to read and write the agent's memory of the interaction history. It appears difficult for the agent to be aware of such modifications in general. We must not forget that amnesia happens for people, and it may happen for robots too, e.g. after a collision. And security by obfuscation can only delay memory hackers, even when considering a natural brain. Another interesting consequence is that deterministic policies cannot be always optimal, on the contrary to the optimal and deterministic AIXI in the dualist framework.

Luke: Do you think the AIXI framework, including the limited but tractable approximations like MC-AIXI, provides a plausible path toward real-world AGI? If not, what do you see as its role in AGI research?

Laurent: Approximating AIXI can be done in very many ways. The main ideas are building/finding good and simple models of the environment, and performing some planning on these models; i.e., it is a model-based approach (by contrast to Q-learning which is model-free for example: it does model the environment, but only learns to predict the expected rewards per action/state). This is a very common approach in reinforcement learning, because some may argue that model-free methods are "blind", in the sense that they don't learn about their environments, they just "know" what to do.  Another important component of AIXI is the interaction history (instead of a state-based observation), and approximations may need to deal appropriately with compressing this history, possibly with loss. Hutter is working on this aspect with feature RL, with nice results. So yes, approximating AIXI can be seen as a very plausible way toward real-world AGI.
Finding computation-efficient approximations is not an easy task, and it will quite probably require a number of neat ideas that will make it feasible, but it's certainly a path worth researching. However, personally, I prefer to think that the agent must learn how to model its environment, which is something deeper than a model-based approach.
Even without considering AIXI approximations, AIXI still is very important for AGI research because it unifies all important properties of cognition, like agency (interaction with an environment), knowledge representation and memory, understanding, reasoning, goals, problem solving, planning and action selection, abstraction, generalization without overfitting, multiple hypotheses, creativity, exploration and curiosity, optimization and utility maximization, prediction, uncertainty, with incremental, on-line, lifelong, continual learning in arbitrarily complex environments, without a restart state, no i.i.d. or stationarity assumption, etc. and does all this in a very simple, elegant and precise manner. I believe that if one does not understand what tour de force AIXI is, one cannot seriously hope to tackle the AGI problem. People tend to think that simple ideas are easy to find because they are easy to read or be explained orally. But they tend to forget that easy to read does not mean easy to grok, and certainly not easy to find! Simplest ideas are the best ones, especially in research.

Luke: You speak of AIXI-related work as a pretty rich subfield with many interesting lines of research to pursue that are related to universal agents. Do you think there are other AGI-related lines of inquiry that are as promising or productive as AIXI-related work? For example Schmidhuber's Gödel machine, the SOAR architecture, etc.?

Laurent: I can't really say much about the cognitive architectures. It looks very difficult to say whether such a design would work correctly autonomously over several decades. It's interesting work with nice ideas, but I can't see what kind of confidence I could have in such designs when regarding long-term AGI. That's why I prefer simple and general approaches, formalized with some convergence proof or a proof of another important property, and that can give you confidence that your design will work for more than a few days ahead.
Regarding the Gödel machine (GM), I do think it's a very nice design, but I have two griefs with it. The first one is that it's currently not sufficiently formalized, so it's difficult to state if and how it really works. The second one is because it's relying on a automated theorem prover. Searching for proofs is extremely complicated: Making a parallel with Levin-search (LS), where given a goal output string (an improvement in the GM), you enumerate programs (propositions in the GM) and run them to see if they output the goal string (search for a proof of improvement in GM). This last part is the problem: in LS, the programs are fast to run, whereas in GM there is an additional search step for each proposition, so this looks very roughly like going from exponential (LS) to double-exponential (GM). And LS is already not really practical.
Theorem proving is even more complicated when you need to prove that there will be an improvement of the system at an unknown future step. Maybe it would work better if the kinds of proofs were limited to some class, for example use simulation of the future steps up to some horizon given a model of the world. These kinds of proofs are easier to check and have a guaranteed termination, e.g. if the model class for the environment is based on Schmidhuber's Speed Prior. But this starts to look pretty much like an approximation of AIXI, doesn't it?
Reinforcement Learning in general is very promising for AGI: Even though most researchers of this field are not directly interested in AGI, they still try to find the most general possible methods while remaining practical. It's far more obvious in RL than in the rest of Machine Learning.
But I think there are many lines of research that have could be pushed toward some kind of AGI level. In machine learning, some fields like Genetic Programming, Inductive Logic Programming, Grammar Induction and −let's get bold− why not Recurrent Deep Neural Networks (probably with an additional short-term memory mechanism of some sort), and possibly other research fields, are all based on some powerful induction schemes that could well lead to thinking machines if researchers in those fields wanted to. Schmidhuber's OOPS is also a very interesting design, based on Levin Search. It is limited in that it cannot truly reach the learning-to-learn level, but could be extended with a true probability distribution over programs, as described by Solomonoff. And there is of course also the neuroscience way of trying to understand or at least model the brain.

Luke: Thanks, Laurent!
The post Laurent Orseau on Artificial General Intelligence appeared first on Machine Intelligence Research Institute.


Five Theses, Using Only Simple Words

A recent xkcd comic described the Saturn V rocket using only the 1000 most frequently used words (in English). The rocket was called "up-goer five," and the liquid hydrogen feed line was the "thing that lets in cold wet air to burn." This inspired a geneticist to make the Up-Goer Five Text Editor, which forces you to use only the 1000 most frequent words. Mental Floss recently collected 18 scientific ideas explained using this restriction.
What does this have to do with MIRI? Well, young philosopher Robby Bensinger has now re-written MIRI's Five Theses using the Up-Goer Five text editor, with amusing results:

Intelligence explosion: If we make a computer that is good at doing hard things in lots of different situations without using much stuff up, it may be able to help us build better computers. Since computers are faster than humans, pretty soon the computer would probably be doing most of the work of making new and better computers. We would have a hard time controlling or understanding what was happening as the new computers got faster and grew more and more parts. By the time these computers ran out of ways to quickly and easily make better computers, the best computers would have already become much much better than humans at controlling what happens.
Orthogonality: Different computers, and different minds as a whole, can want very different things. They can want things that are very good for humans, or very bad, or anything in between. We can be pretty sure that strong computers won't think like humans, and most possible computers won't try to change the world in the way a human would.
Convergent instrumental goals: Although most possible minds want different things, they need a lot of the same things to get what they want. A computer and a human might want things that in the long run have nothing to do with each other, but have to fight for the same share of stuff first to get those different things.
Complexity of value: It would take a huge number of parts, all put together in just the right way, to build a computer that does all the things humans want it to (and none of the things humans don't want it to).
Fragility of value: If we get a few of those parts a little bit wrong, the computer will probably make only bad things happen from then on. We need almost everything we want to happen, or we won't have any fun.

That is all. You're welcome.
The post Five Theses, Using Only Simple Words appeared first on Machine Intelligence Research Institute.

Five Theses, Using Only Simple Words


How effectively can we plan for future decades? (initial findings)

MIRI aims to do research now that increases humanity's odds of successfully managing important AI-related events that are at least a few decades away. Thus, we'd like to know: To what degree can we take actions now that will predictably have positive effects on AI-related events decades from now? And, which factors predict success and failure in planning for decades-distant events that share important features with future AI events?
Or, more generally: How effectively can humans plan for future decades? Which factors predict success and failure in planning for future decades?
To investigate these questions, we asked Jonah Sinick to examine historical attempts to plan for future decades and summarize his findings. We pre-committed to publishing our entire email exchange on the topic (with minor editing), just as Jonah had done previously with GiveWell on the subject of insecticide-treated nets. The post below is a summary of findings from our full email exchange (.pdf) so far.
We decided to publish our initial findings after investigating only a few historical cases. This allows us to gain feedback on the value of the project, as well as suggestions for improvement, before continuing. It also means that we aren't yet able to draw any confident conclusions about our core questions.
The most significant results from this project so far are:

Jonah's initial impressions about The Limits to Growth (1972), a famous forecasting study on population and resource depletion, were that its long-term predictions were mostly wrong, and also that its authors (at the time of writing it) didn't have credentials that would predict forecasting success. Upon reading the book, its critics, and its defenders, Jonah concluded that many critics and defenders had  seriously misrepresented the book, and that the book itself exhibits high epistemic standards and does not make significant predictions that turned out to be wrong.
Svante Arrhenius (<PHONE>) did a surprisingly good job of climate modeling given the limited information available to him, but he was nevertheless wrong about two important policy-relevant factors. First, he failed to predict how quickly carbon emissions would increase. Second, he predicted that global warming would have positive rather than negative humanitarian impacts. If more people had taken Arrhenius' predictions seriously and burned fossil fuels faster for humanitarian reasons, then today's scientific consensus on the effects of climate change suggests that the humanitarian effects would have been negative.
In retrospect, Norbert Weiner's concerns about the medium-term dangers of increased automation appear naive, and it seems likely that even at the time, better epistemic practices would have yielded substantially better predictions.
Upon initial investigation, several historical cases seemed unlikely to shed substantial light on our  core questions: Norman Rasmussen's analysis of the safety of nuclear power plants, Leo Szilard's choice to keep secret a patent related to nuclear chain reactions, Cold War planning efforts to win decades later, and several cases of "ethically concerned scientists."
Upon initial investigation, two historical cases seemed like they might shed light on our  core questions, but only after many hours of additional research on each of them: China's one-child policy, and the Ford Foundation's impact on India's 1991 financial crisis.
We listed many other historical cases that may be worth investigating.

The project has also produced a chapter-by-chapter list of some key lessons from Nate Silver's The Signal and the Noise, available here.
Further details are given below. For sources and more, please see our full email exchange (.pdf).

The Limits to Growth
In his initial look at The Limits to Growth (1972), Jonah noted that the authors were fairly young at the time of writing (the oldest was 31), and they lacked credentials in long-term forecasting. Moreover, it appeared that Limits to Growth predicted a sort of doomsday scenario – ala Erlich's The Population Bomb (1968) – that had failed to occur. In particular, it appeared that Limits to Growth had failed to appreciate Julian Simon's point that other resources would substitute for depleted resources.
Upon reading the book, Jonah found that:

The book avoids strong, unconditional claims. Its core claim is that if exponential growth of resource usage continues, then there will likely be a societal collapse by 2100.
The book was careful to qualify its claims, and met high epistemic standards. Jonah wrote: "The book doesn't look naive even in retrospect, which is impressive given that it was written 40 years ago. "
The authors discuss substitutability at length in chapter 4.
The book discusses mitigation at a theoretical level, but doesn't give explicit policy recommendations, perhaps because the issues involved were too complex.

Svante Arrhenius
Derived more than a century ago, Svante Arrhenius' equation for how the Earth's temperature varies as a function of concentration of carbon dioxide is the same equation used today. But while Arrhenius' climate modeling was impressive given the information available to him at the time, he failed to predict (by a large margin) how quickly fossil fuels would be burned. He also predicted that global warming would have positive humanitarian effects, but based on our current understanding, the expected humanitarian effects seem negative.
Arrhenius's predictions were mostly ignored at the time, but had people taken them seriously and burned fossil fuels more quickly, the humanitarian effects would probably have been negative.

Norbert Weiner
As Jonah explains, Norbert Weiner (<PHONE>) "believed that unless countermeasures were taken, automation would render low skilled workers unemployable. He believed that this would precipitate an economic crisis far worse than that of the Great Depression." Nearly 50 years after his death, this doesn't seem to have happened much, though it may eventually happen.
Jonah's impression is that Weiner had strong views on the subject, doesn't seem to have updated much in response to incoming evidence, and seems to have relied to heavily on what Berlin (1953) and Tetlock (2005) described as "hedgehog" thinking: "the fox knows many things, but the hedgehog knows one big thing."

Some historical cases that seem unlikely to shed light on our questions
Rasmussen (1975) is a probabilistic risk assessment of nuclear power plants, written before any nuclear power plant disasters had occurred. However, Jonah concluded that this historical case wasn't very relevant to our specific questions about taking actions useful for decades-distant AI outcomes, in part because the issue is highly domain specific, and because the report makes a large number of small predictions rather than a few salient predictions.
In 1936, Leó Szilárd assigned his chain reaction patent in a way that ensured it would be kept secret from the Nazis. However, Jonah concluded:
I think that this isn't a good example of a nontrivial future prediction. The destructive potential seems pretty obvious – anything that produces a huge amount of concentrated energy can be used in a destructive way. As for the Nazis, Szilard was himself Jewish and fled from the Nazis, and it seems pretty obvious that one wouldn't want a dangerous regime to acquire knowledge that has destructive potential. It would be more impressive if the early developers of quantum mechanics had kept their research secret on account of dimly being aware of the possibility of destructive potential, or if Szilard had filed his patent secretly in a hypothetical world in which the Nazi regime was years away.
Jonah briefly investigated Cold War efforts aimed at winning the war decades later, but concluded that it was "too difficult to tie these efforts to war outcomes."
Jonah also investigated Kaj Sotala's A brief history of ethically concerned scientists. Most of the historical cases cited there didn't seem relevant to this project. Many cases involved "scientists concealing their discoveries out of concern that they would be used for military purposes," but this seems to be an increasingly irrelevant sort of historical case, since science and technology markets are now relatively efficient, and concealing a discovery rarely delays progress for very long (e.g. see Kelly 2011). Other cases involved efforts to reduce the use of dangerous weapons for which the threat was imminent during the time of the advocacy. There may be lessons among these cases, but they appear to be of relatively weak relevance to our current project.

Some historical cases that might shed light on our questions with much additional research
Jonah performed an initial investigation of the impacts of China's one-child policy, and concluded that it would take many, many hours of research to determine both the sign and the magnitude of the policy's impacts.
Jonah also investigated a case involving the Ford Foundation. In a conversation with GiveWell, Lant Pritchett said:
[One] example of transformative philanthropy is related to India's recovery from its economic crisis of 1991. Other countries had previously had similar crises and failed to implement good policies that would have allowed them to recover from their crises. By way of contrast, India implemented good policies and recovered in a short time frame. Most of the key actors who ensured that India implemented the policies that it did were influenced by a think tank established by the Ford Foundation ten years before the crisis. The think tank exposed Indians to relevant ideas from the developed world about liberalization. The difference between (a) India's upward economic trajectory and (b) what its upward economic trajectory would have been if it had been unsuccessful in recovering from the 1991 crisis is in the trillions of dollars. As such, the Ford Foundation's investment in the think tank had a huge impact. For the ten years preceding the crisis, it looked like the think tank was having no impact, but it turned out to have a huge impact.
Unfortunately, Jonah was unable to find any sources or contacts that would allow him to check whether this story is true.

Other historical cases that might be worth investigating
Historical cases we identified but did not yet investigate include:

Eric Drexler's early predictions about the feasibility and likely effects of nanotechnology.
The Asilomar conference on recombinant DNA
Efforts to detect asteroids before they threaten Earth
The Green Revolution
The modern history of cryptography
Early efforts to mitigate global warming
Possible deliberate long term efforts to produce scientific breakthroughs (the transistor? the human genome?)
Rachel Carson's Silent Spring (1962)
Paul Ehrlich's The Population Bomb (1968)
The Worldwatch Institute's State of the World reports (since 1984)
The WCED's Our Common Future (1987)

The post How effectively can we plan for future decades? (initial findings) appeared first on Machine Intelligence Research Institute.


The Hanson-Yudkowsky AI-Foom Debate is now available as an eBook!

In late 2008, economist Robin Hanson and AI theorist Eliezer Yudkowsky conducted an online debate about the future of artificial intelligence, and in particular about whether generally intelligent AIs will be able to improve their own capabilities very quickly (a.k.a. "foom"). James Miller and Carl Shulman also contributed guest posts to the debate.
The debate is now available as an eBook in various popular formats (PDF, EPUB, and MOBI). It includes:

the original series of blog posts,
a transcript of a 2011 in-person debate between Hanson and Yudkowsky on this subject,
a summary of the debate written by Kaj Sotala, and
a 2013 technical report on AI takeoff dynamics ("intelligence explosion microeconomics") written by Yudkowsky.

Comments from the authors are included at the end of each chapter, along with a link to the original post.
Head over to intelligence.org/ai-foom-debate/ to download a free copy.
The post The Hanson-Yudkowsky AI-Foom Debate is now available as an eBook! appeared first on Machine Intelligence Research Institute.


Stephen Hsu on Cognitive Genomics

Stephen Hsu is Vice-President for Research and Graduate Studies and Professor of Theoretical Physics at Michigan State University. Educated at Caltech and Berkeley, he was a Harvard Junior Fellow and held faculty positions at Yale and the University of Oregon. He was also founder of SafeWeb, an information security startup acquired by Symantec. Hsu is a scientific advisor to BGI and a member of its Cognitive Genomics Lab.

Luke Muehlhauser: I'd like to start by familiarizing our readers with some of the basic facts relevant to the genetic architecture of cognitive ability, which I've drawn from the first half of a presentation you gave in February 2013:

The human genome consists of about 3 billion base pairs, but humans are very similar to each other, so we only differ from each other on about 3 million of these base pairs.
Because there's so much repetition, we could easily store the entire genome of every human on earth (~3mb per genome, compressed).
Scanning someone's SNPs costs about $200; scanning their entire genome costs $1000 or more.
But, genotyping costs are falling so quickly that SNPs may be irrelevant soon, as it'll be simpler and cheaper to just sequence entire genomes.
To begin to understand the genetic architecture of cognitive ability, we can compare it to the genetic architecture of height, since the genetic architectures of height and cognitive ability are qualitatively the same.
For example, (1) height and cognitive ability are relatively stable and reliable traits (in adulthood), meaning that if you measure a person's height or cognitive ability at multiple times you'll get roughly the same result each time, (2) height and cognitive ability are valid traits, in that they "measure something real" that is predictive of various life outcome measures like income, (3) both height and cognitive ability are highly heritable, and (4) both height and cognitive ability are highly polygenic, meaning that many different genes contribute to height and cognitive ability.
All cognitive observables — e.g. vocabulary, digit recall (short term memory), ability to solve math puzzles, spatial rotation ability, cognitive reaction time — appear to be positively correlated. Because of this, we can (lossily) compress the data for how a person scores on different cognitive tests to a single number, which we call IQ, and this single number is predictive of their scores on all cognitive tests, and also life outcome measures like income, educational attainment, job performance, and mortality.
This contradicts some folk wisdom. E.g. parents often believe that "Johnny's good at math, so he's probably not going to be good with words." But in fact, the data show that math skill is quite predictive of verbal skill, because (roughly) all cognitive abilities are positively correlated.
By convention, IQ is normally distributed in the population with a mean at 100 and a standard deviation of 15.
Culturally neutral cognitive tests like progressive matrices are very tightly correlated (0.9) with IQ. So you can estimate someone's IQ (and hence their verbal ability, spatial rotation ability, short term memory, cognitive reaction time, etc.) pretty well using only one test like Raven's progressive matrices.
It's very difficult to raise one's score on these cognitive tests with training. In large studies, it looks like thousands of dollars worth of training can raise your score by a small fraction of the standard deviation.
Additional IQ points do appear to "matter" — even above, say, IQ 145. E.g. the mean IQ of eminent scientists (IQ 160) is much higher than that of average PhDs (IQ 130). Also, in a longitudinal study of children identified as gifted at age 13, the "1 in 10,000"-level children had significantly better life outcomes than the "1 in 100"-level children, even though they generally all received "gifted child" development paths.

One source of details and references for most of this is The Cambridge Handbook of Intelligence.
Before we continue, Stephen, do you have any corrections or clarifications you'd like to make about my summary, or additional sources that you'd like to recommend to our readers?

Stephen Hsu: A couple of comments on the summary, which is excellent:

Raven's correlation might not be as high as 0.9 with overall IQ, it might actually be 0.8 or so. These numbers fluctuate around depending on the study. In general two tests might be considered valid "IQ tests" if they correlate at > 0.75 or so with g. This is the case with most standardized tests like ACT, SAT, GRE, etc.
Mean IQ of participants in the Roe study was quite high, but I doubt that the average among eminent scientists (averaging over all fields) is 160; probably a bit lower like 145. In any case the Roe and SMPY data are sufficient to suggest nontrivial returns to IQ above 130 in STEM.

It seems you understood my talk perfectly well. The answers to your questions may already be in there, but I'm happy to discuss and clarify further.

Luke: Have we identified any genes that are (with high confidence) associated with cognitive ability? What can our history of identifying genes associated with other polygenic traits (e.g. height) tell us about our prospects for identifying genes associated with cognitive ability?

Stephen: Recently the results of a massive GWAS for genes associated with educational attainment were published in Science. Some of the researchers in this large collaboration are reluctant to openly state that the hits are associated with cognitive ability (as opposed to, say, Conscientiousness, which would also positively impact educational success). But if you read the paper carefully you can see that there is good evidence that the alleles are actually associated with cognitive ability (g or IQ).
At the link above you can find a historical graph:

This graph displays the number of GWAS hits versus sample size for height, BMI, etc. Once the minimal sample size to discover the alleles of largest impact (large MAF, large effect size) is exceeded, one generally expects a steady accumulation of new hits at lower MAF / effect size. I expect the same sort of progress for g. (MAF = Minor Allele Frequency. Variants that are common in the population are easier to detect than rare variants.)
We can't predict the sample size required to obtain most of the additive variance for g (this depends on the details of the distribution of alleles), but I would guess that about a million genotypes together with associated g scores will suffice. When, exactly, we will reach this sample size is unclear, but I think most of the difficulty is in obtaining the phenotype data. Within a few years, over a million people will have been genotyped, but probably we will only have g scores for a small fraction of the individuals.

Luke: Could you describe for us the goals and methods of the work you're currently doing with BGI?

Stephen: The goal of our cognitive genomics project at BGI is to understand the genetic architecture of human cognition. There are obviously many potential applications of this work, in areas ranging from deep human history (evolution) to drug discovery to genetic engineering. But my primary interest is intellectual.
The methods are straightforward: obtain genotype and phenotype data and look for statistical associations (GWAS). More specifically, we want to determine the parameters of a polygenic model relating genotype to phenotype. (This as yet undetermined set of "fundamental constants" is one of the most interesting few megabytes of information in the biological world.) The leading term in this model is linear (meaning we are guaranteed a certain amount of progress from simple techniques), but eventually we will be interested in nonlinear corrections (epistasis, gene-gene interactions, dominance, etc.) as well.
We started out by looking for high g individuals because, as outliers, they produce more statistical power per dollar of sequencing. The cost of sequencing is still our primary constraint, and will be for at least a few more years. For example, the cost to sequence our 2000 high g volunteers is well into the millions of dollars. I also felt, given my background, that I had reasonable insight into where to find and how to recruit volunteers from the high g tail.
Ultimately, I hope that various genomics labs around the world will collaborate to produce a public data repository with g as one of the phenotype variables.
Link:
International partners describe global alliance to enable secure sharing of genomic and clinical data

Luke: How feasible do you think "iterated embryo selection" will be, over the next several decades, for the amplification of cognitive abilities via genetic selection?
Background for our readers: iterated embryo selection is a plausible future technology that could allow strong genetic selection for intelligence without needing to wait 15-20 years between generations. It was first described in detail in the FAQ for MIRI's The Uncertain Future project (see Rayhawk et al. 2009), was later described in a book (Miller 2012), and was finally published in a journal in Sparrow (2013).

Stephen: I have no particular insight into specific challenges related to producing gametes from pluripotent stem cells. It's not my area of expertise. However, I am confident that genomic selection for traits such as g will be possible. I would be surprised if, after analyzing millions of genotype-phenotype pairs, we were not able to produce a predictive model that captures, say, 50% of variance in g. That means, roughly, we might be able to predict g from genotype with standard error of somewhat less than a population standard deviation (e.g., 10 IQ points; note I don't think the real world "meaning" of g is better defined than within an error of this size). This means that selection on g can proceed relatively efficiently, assuming the basic reproductive technologies are under control.
I think there is good evidence that existing genetic variants in the human population (i.e., alleles affecting intelligence that are found today in the collective world population, but not necessarily in a single person) can be combined to produce a phenotype which is far beyond anything yet seen in human history. This would not surprise an animal or plant breeder — experiments on corn, cows, chickens, drosophila, etc. have shifted population means by many standard deviations (e.g., +30 SD in the case of corn).
Let me add that, in my opinion, each society has to decide for itself (e.g. through democratic process) whether it wants to legalize or forbid activities that amount to genetic engineering. Intelligent people can reasonably disagree as to whether such activity is wise.
Links:
"Only he was fully awake"
Maxwell's Demon and genetic engineering
Epistasis vs additivity
Deleterious variants affecting traits that have been under selection are rare and of small effect

Luke: Work on the genetics of cognitive ability tends to be more controversial than work on the genetics of, say, height. Why do you think that is? Has your work, or the work of your colleagues, been made more difficult because of such issues?

Stephen: Given our difficult history with race there is an understandable discomfort with the idea that cognitive ability is strongly influenced by genetics. In the worst case, it might be found that historically isolated populations of humans differ in their average genetic capacities for cognition, due to variation in allele frequencies. Let me stress that at the moment our understanding of the genetics of intelligence is far too preliminary to reach a firm conclusion on this issue.
At the extremes, there are some academics and social activists who violently oppose any kind of research into the genetics of cognitive ability. Given that the human brain — its operation, construction from a simple genetic blueprint, evolutionary history — is one of the great scientific mysteries of the universe, I cannot understand this point of view.

Luke: What do you think a truly superior human intelligence would be like?

Stephen: I think we already have some hints in this direction. Take the case of John von Neumann, widely regarded as one of the greatest intellects in the 20th century, and a famous polymath. He made fundamental contributions in mathematics, physics, nuclear weapons research, computer architecture, game theory and automata theory.
In addition to his abstract reasoning ability, von Neumann had formidable powers of mental calculation and a photographic memory. In my opinion, genotypes exist that correspond to phenotypes as far beyond von Neumann as he was beyond a normal human.
I have known a great many intelligent people in my life. I knew Planck, von Laue and Heisenberg. Paul Dirac was my brother in law; Leo Szilard and Edward Teller have been among my closest friends; and Albert Einstein was a good friend, too. But none of them had a mind as quick and acute as Jansci [John] von Neumann. I have often remarked this in the presence of those men and no one ever disputed me.
— Nobel Laureate Eugene Wigner
You know, Herb, how much faster I am in thinking than you are. That is how much faster von Neumann is compared to me.
— Nobel Laureate Enrico Fermi to his former PhD student Herb Anderson.
One of his remarkable abilities was his power of absolute recall. As far as I could tell, von Neumann was able on once reading a book or article to quote it back verbatim; moreover, he could do it years later without hesitation. He could also translate it at no diminution in speed from its original language into English. On one occasion I tested his ability by asking him to tell me how The Tale of Two Cities started. Whereupon, without any pause, he immediately began to recite the first chapter and continued until asked to stop after about ten or fifteen minutes.
— Herman Goldstine, mathematician and computer pioneer.
I always thought Von Neumann's brain indicated that he was from another species, an evolution beyond man,
— Nobel Laureate Hans A. Bethe.
Links:
Wikipedia: John von Neumann
The differences are enormous
"Only he was fully awake"

Luke: Thanks, Stephen!
The post Stephen Hsu on Cognitive Genomics appeared first on Machine Intelligence Research Institute.

Stephen Hsu on Cognitive Genomics


MIRI's November 2013 Workshop in Oxford

From November 23-29, MIRI will host another Workshop on Logic, Probability, and Reflection, for the first time in Oxford, UK.
Participants will investigate problems related to reflective agents, probabilistic logic, and priors over logical statements / the logical omniscience problem.
Participants confirmed so far include:

Stuart Armstrong (Oxford)
Mihaly Barasz (Google)
Catrin Campbell-Moore (LMU Munich)
Daniel Dewey (Oxford)
Benja Fallenstein (U Bristol)
Jacob Hilton (U Leeds)
Ramana Kumar (Cambridge)
Jan Leike (U Freiburg)
Bas Steunebrink (IDSIA)
Gregory Wheeler (LMU Munich)
Eliezer Yudkowsky (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it's not too late to apply! And even if this workshop doesn't fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).
The post MIRI's November 2013 Workshop in Oxford appeared first on Machine Intelligence Research Institute.

MIRI’s November 2013 Workshop in Oxford


Transparency in Safety-Critical Systems

In this post, I aim to summarize one common view on AI transparency and AI reliability. It's difficult to identify the field's "consensus" on AI transparency and reliability, so instead I will present a common view so that I can use it to introduce a number of complications and open questions that (I think) warrant further investigation.
Here's a short version of the common view I summarize below:
Black box testing can provide some confidence that a system will behave as intended, but if a system is built such that it is transparent to human inspection, then additional methods of reliability verification are available. Unfortunately, many of AI's most useful methods are among its least transparent. Logic-based systems are typically more transparent than statistical methods, but statistical methods are more widely used. There are exceptions to this general rule, and some people are working to make statistical methods more transparent.

The value of transparency in system design
Nusser (2009) writes:
…in the field of safety-related applications it is essential to provide transparent solutions that can be validated by domain experts. "Black box" approaches, like artificial neural networks, are regarded with suspicion – even if they show a very high accuracy on the available data – because it is not feasible to prove that they will show a good performance on all possible input combinations.
Unfortunately, there is often a tension between AI capability and AI transparency. Many of AI's most powerful methods are also among its least transparent:
Methods that are known to achieve a high predictive performance — e.g. support vector machines (SVMs) or artificial neural networks (ANNs) — are usually hard to interpret. On the other hand, methods that are known to be well-interpretable — for example (fuzzy) rule systems, decision trees, or linear models — are usually limited with respect to their predictive performance.1
But for safety-critical systems — and especially for AGI — it is important to prioritize system reliability over capability. Again, here is Nusser (2009):
strict requirements [for system transparency] are necessary because a safety-related system is a system whose malfunction or failure can lead to serious consequences — for example environmental harm, loss or severe damage of equipment, harm or serious injury of people, or even death. Often, it is impossible to rectify a wrong decision within this domain.
The special need for transparency in AI has also been stressed by many others,2 including Boden (1977):
Members of the artificial intelligence community bear an ominous resemblance to… the Sorcerer's Apprentice. The apprentice learnt just enough magic… to save himself the trouble of performing an onerous task, but not quite enough to stop the spellbound buckets and brooms from flooding the castle…
[One question I shall ask is] whether there are any ways of writing programs that would tend to keep control in human hands… [For one thing,] programs should be intelligible and explicit, so that "what is going on" is not buried in the code or implicitly embodied in procedures whose aim and effect are obscure.

A spectrum from black box to transparent
Non-transparent systems are sometimes called "black boxes":
a black box is a device, system or object which can be viewed in terms of its input, output and transfer characteristics without any knowledge of its internal workings. Its implementation is "opaque" (black). Almost anything might be referred to as a black box: a transistor, an algorithm, or the human mind.
…[And] in practice some [technically transparent] systems are so complex that [they] might as well be [black boxes].3
The human brain is mostly a black box. We can observe its inputs (light, sound, etc.), its outputs (behavior), and some of its transfer characteristics (swinging a bat at someone's eyes often results in ducking or blocking behavior), but we don't know very much about how the brain works. We've begun to develop an algorithmic understanding of some of its functions (especially vision), but only barely.4
Many contemporary AI methods are effectively black box methods. As Whitby (1996) explains, the safety issues that arise in "GOFAI" (e.g. search-based problem solvers and knowledge-based systems) "are as nothing compared to the [safety] problems which must be faced by newer approaches to AI… Software that uses some sort of neural net or genetic algorithm must face the further problem that it seems, often almost by definition, to be 'inscrutable'. By this, I mean that… we can know that it works and test it over a number of cases but we will not in the typical case ever be able to know exactly how."
Other methods, however, are relatively transparent, as we shall see below.
This post cannot survey the transparency of all AI methods; there are too many. Instead, I will focus on three major "families" of AI methods.

Examining the transparency of three families of AI methods

Machine learning
Machine learning is perhaps the largest and most active subfield in AI, encompassing a wide variety of methods by which machines learn from data. For an overview of the field, see Flach (2012). For a quick video intro, see here.
Unfortunately, machine learning methods tend not to be among the most transparent methods.5 Nusser (2009) explains:
machine learning approaches are regarded with suspicion by domain experts in safety-related application fields because it is often infeasible to sufficiently interpret and validate the learned solutions.
For now, let's consider one popular machine learning method in particular: artificial neural networks (ANNs). (For a concise video introduction, see here.) As Rodvold (1999) explains, ANNs are typically black boxes:
the intelligence of neural networks is contained in a collection of numeric synaptic weights, connections, transfer functions, and other network definition parameters. In general, inspection of these quantities yields little explicit information to enlighten developers as to why a certain result is being produced.
Also, Kurd (2005):
it is common for typical ANNs to be treated as black-boxes… because ANN behaviour is scattered across its weights and links with little meaning to an observer. As a result of this unstructured and unorganised representation of behaviour, it is often not feasible to completely understand and predict their function and operation… The interpretation problems associated with ANNs impede their use in safety critical contexts…6
Deep learning is another popular machine learning technique. It, too, tends to be non-transparent — like ANNs, deep learning methods were inspired by how parts of the brain work, in particular the visual system.7
Some machine learning methods are more transparent than others. Bostrom & Yudkowsky (2013) explain:
If [a] machine learning algorithm is based on a complicated neural network… then it may prove nearly impossible to understand why, or even how, the algorithm [made its judgments]. On the other hand, a machine learner based on decision trees or Bayesian networks is much more transparent to programmer inspection (Hastie et al. 2001), which may enable an auditor to discover [why the algorithm made the judgments it did].8
Moreover, recent work has attempted to make some machine learning methods more transparent, and thus perhaps more suitable for safety-critical applications. For example, Taylor (2005) suggests methods for extracting rules (which refer to human-intuitive concepts) from neural networks, so that researchers can perform formal safety analyses of the extracted rules. These methods are still fairly primitive, and are not yet widely applicable or widely used, but further research could make these methods more useful and popular.9

Evolutionary algorithms
Evolutionary algorithms (EAs) are often categorized as a machine learning method, but here they will be considered separately. EAs use methods inspired by evolution to produce candidate solutions to problems. For example, watch this video of software robots evolving to "walk" quickly.
Because evolutionary algorithms use a process of semi-random mutation and recombination to produce candidate solutions, complex candidate solutions tend not to be transparent — just like the evolultionarily-produced brain. Mitchell (1998), p. 40 writes:
Understanding the results of [genetic algorithm] evolution is a general problem — typically the [genetic algorithm] is asked to find [candidate solutions] that achieve high fitness but is not told how that high fitness is to be attained. One could say that this is analogous to the difficulty biologists have in understanding the products of natural evolution (e.g., us)… In many cases… it is difficult to understand exactly how an evolved high−fitness [candidate solution] works. In genetic programming, for example, the evolved programs are often very long and complicated, with many irrelevant components attached to the core program performing the desired computation. It is usually a lot of work — and sometimes almost impossible — to figure out by hand what that core program is.
Fleming & Purshouse (2002) add:
Mission-critical and safety-critical applications do not appear, initially, to be favourable towards EA usage due to the stochastic nature of the evolutionary algorithm. No guarantee is provided that the results will be of sufficient quality for use on-line.

Logical methods
Logical methods in AI are implemented widely in safety-critical applications (e.g. medicine), but see far less application in general compared to machine learning methods.
In a logic-based AI, the AI's knowledge and its systems for reasoning are written out in logical statements. These statements are typically hand-coded, and the meaning of each statement has a precise meaning determined by the axioms of the logical system being used (e.g. first-order logic). Russell & Norvig (2009), the leading AI textbook, describes the logical approach to AI in chapter 7. It describes a popular application of logical AI, called "classical planning," in chapter 10. Also see Thomason (2012) and Minker (2000).
Galliers (1988, p. 88-89) explains the transparency advantages of logic-based methods in AI:
A theory expressed as a set of logical axioms is evident; it is open to examination. This assists the process of determining whether any parts of the theory are inconsistent, or do not behave as had been anticipated when they were expressed in English… Logics are languages with precise semantics [and therefore] there can be no ambiguities of interpretation… By expressing the properties of agents… as logical axioms and theorems… the theory is transparent; properties, interrelationships and inferences are open to examination… This contrasts with the use of computer code [where] it is frequently the case that computer systems concerned with… problem-solving are in fact designed such that properties of the interacting agents are implicit properties of the entire system, and it is impossible to investigate the role or effects of any individual aspect.10
Another transparency advantage of logical methods in AI comes from logic languages' capacity to represent different kinds of machines, including machines that can reflect on themselves and the reasons for their beliefs. For example, they can pass assumptions around with each datum. E.g. see the "domino" agent in Fox & Das (2000).
Moreover, some logic-based approaches are amenable to formal methods, for example formal verification: mathematically proving that a system will perform correctly with respect to a formal specification.11 Formal methods complement empirical testing of software, e.g. by identifying "corner bugs" that are difficult to find when using empirical methods only — see Mitra 2008.
Formal verification is perhaps best known for its use in verifying hardware components (especially since the FDIV bug that cost Intel $500 million), but it is also used to verify a variety of software programs (in part or in whole), including flight control systems (Miller et al. 2005), rail control systems (Platzer & Quesel 2009), pacemakers (Tuan et al. 2010), compilers (Leroy 2009), operating system kernels (Andronick 2011), multi-agent systems (Raimondi 2006), outdoor robots (Proetzsch et al. 2007), and swarm robotics (Dixon et al. 2012).
Unfortunately, formal methods face severe limitations. Fox (1993) explains:
there are severe limitations on the capability of formal design techniques to completely prevent hazardous situations from arising. Current formal design methods are difficult to use and time-consuming, and may only be practical for relatively modest applications. Even if we reserve formal techniques for the safety-critical elements of the system we have seen that the soundness guaranteed by the techniques can only be as good as the specifier's ability to anticipate the conditions and possible hazards that can hold at the time of use… These problems are difficult enough for 'closed systems' in which the designer can be confident, in principle, of knowing all the parameters which can affect system performance… Unfortunately all systems are to a greater or lesser extent 'open'; they operate in an environment which cannot be exhaustively monitored and in which unpredictable events will occur. Furthermore, reliance on specification and verification methods assumes that the operational environment will not compromise the correct execution of software. In fact of course software errors can be caused by transient faults causing data loss or corruption; user errors; interfacing problems with external systems (such as databases and instruments); incompatibilities between software versions; and so on.12
Bowen & Hinchey (1995) concur:
There are many… areas where, although possible, formalization is just not practical from a resource, time, or financial aspect. Most successful formal methods projects involve the application of formal methods to critical portions of system development. Only rarely are formal methods, and formal methods alone, applied to all aspects of system development. Even within the CICS project, which is often cited as a major application of formal methods… only about a tenth of the entire system was actually subjected to formal techniques…
[We suggest] the following maxim: System development should be as formal as possible, but not more formal.
For more on the use of formal methods for AI safety, see Rushby & Whitehurst (1989); Bowen & Stavridou (1993); Harper (2000); Spears (2006); Fischer et al (2013).13

Some complications and open questions
The common view of transparency and AI safety articulated above suggests an opportunity for differential technological development. To increase the odds that future AI systems are safe and reliable, we can invest disproportionately in transparent AI methods, and also in techniques for increasing the transparency of typically opaque AI methods.
But this common view comes with some serious caveats, and some difficult open questions. For example:

How does the transparency of a method change with scale? A 200-rules logical AI might be more transparent than a 200-node Bayes net, but what if we're comparing 100,000 rules vs. 100,000 nodes? At least we can query the Bayes net to ask "what it believes about X," whereas we can't necessarily do so with the logic-based system.
Do the categories above really "carve reality at its joints" with respect to transparency? Does a system's status as a logic-based system or a Bayes net reliably predict its transparency, given that in principle we can use either one to express a probabilistic model of the world?
How much of a system's transparency is "intrinsic" to the system, and how much of it depends on the quality of the user interface used to inspect it? How much of a "transparency boost" can different kinds of systems get from excellently designed user interfaces?14

Acknowledgements
My thanks to John Fox, Jacob Steinhardt, Paul Christiano, Carl Shulman, Eliezer Yudkowsky, and others for their helpful feedback.

Quote from Nusser (2009). Emphasis added. The original text contains many citations which have been removed in this post for readability. Also see Schultz & Cronin (2003), which makes this point by graphing four AI methods along two axes: robustness and transparency. Their graph is available here. In their terminology, a method is "robust" to the degree that it is flexible and useful on a wide variety of problems and data sets. On the graph, GA means "genetic algorithms," NN means "neural networks," PCA means "principal components analysis," PLS means "partial least squares," and MLR means "multiple linear regression." In this sample of AI methods, the trend is clear: the most robust methods tend to be the least transparent. Schultz & Cronin graphed only a tiny sample of AI methods, but the trend holds more broadly.I will share some additional quotes on the importance of transparency in intelligent systems. Kröske et al. (2009) write that, to trust a machine intelligence, "human operators need to be able to understand [its] reasoning process and the factors that precipitate certain actions." Similarly, Fox (1993) writes: "Many branches of engineering have moved beyond purely empirical testing [for safety]… because they have established strong design theories… The consequence is that designers can confidently predict failure modes, performance boundary conditions and so forth before the systems are implemented… A promising approach to [getting these benefits in AI] may be to use well-defined specification languages and verification procedures. Van Harmelen & Balder (1992) [list some] advantages of using formal languages… [including] the removal of ambiguity… [and] the ability to derive properties of the design in the absence of an implementation." In their preface, Fox & Das (2000) write: "Our first obligation is to try to ensure that the designs of our systems are sound. We need to ask not only 'do they work?' but also 'do they work for good reasons?' Unfortunately, conventional software design is frequently ad hoc, and AI software design is little better and possibly worse… Consequently, we place great emphasis on clear design principles, strong mathematical foundations for these principles, and effective development tools that support and verify the integrity of the system… We are creating a powerful technology [AI], possibly more quickly than we think, that has unprecedented potential to create havoc as well as benefit. We urge the community to embark on a vigorous discussion of the issues and the creation of an explicit 'safety culture' in the field."Emphasis added. The first paragraph is from Wikipedia's black box page; the second paragraph is from its white box page. The term "grey box" is sometimes used to refer to methods that are intermediate in transparency between "fully black box" and "fully transparent" methods: see e.g. Sohlberg (2003).Thus, if we could build a whole brain emulation today, it would also be mostly a black box system, even though all its bits of information would be stored in a computer and accessible to database search tools and so on. But we'll probably make lots of progress in cognitive neuroscience before WBE is actually built, and a also working WBE would probably enable quick advances in cognitive neuroscience, and therefore the human brain would rapidly become more transparent to us.For more discussion of how machine learning can be used for relatively "transparent" ends, for example to learn the structure of a Bayesian network, see Murphy (2012), ch. 26.Li & Peng (2006) make the same point: "conventional neural networks… lack transparency, as their activation functions (AFs) and their associated neural parameters bear very little physical meaning." See also Woodman et al. (2012)'s comments on this issue in the context of personal robotics: "Among the requirements of autonomous robots… is a certain degree of robustness. This means being able to handle errors and to continue operation during abnormal conditions… in a dynamic environment, the robot will frequently find itself in a wide range of previously unseen situations. To date, the majority of research in this area has addressed this issue by using learning algorithms, often implemented as artificial neural networks (ANNs)… However, as Nehmzow et al. (2004) identify, these implementations, although seemingly effective, are difficult to analyse due to the inherent opacity of connection-based algorithms. This means that it is difficult to produce an intelligible model of the system structure that could be used in safety analysis."Murphy (2012), p. 995, writes that "when we look at the brain, we seem many levels of processing. It is believed that each level is learning features or representations at increasing levels of abstraction. For example, the standard model of the visual cortex… suggests that (roughly speaking) the brain first extracts edges, then patches, then surfaces, then objects, etc… This observation has inspired a recent trend in machine learning known as deep learning… which attempts to replicate this kind of architecture in a computer."It is generally accepted that Bayesian networks are more transparent than ANNs, but this is only true up to a point. A Bayesian network with hundreds of nodes that are not associated with human-intuitive concepts is not necessarily any more transparent than a large ANN.For an overview of this work, see Nusser (2009), section 2.2.3. Also see Pulina & Tacchella (2011). Finally, Ng (2011), sec. 4, notes that we can get a sense of what function an ANN has learned by asking what which inputs would maximize the activation of particular nodes. In his example, Ng uses this technique to visualize which visual features have been learned by a sparse autoencoder trained on image data.Wooldridge (2003) concurs, writing that "Transparency is another advantage [of logical approaches]."For recent overviews of formal methods in general, see Bozzano & Villafiorita (2010), Woodcock et al. (2009); Gogolla (2006); Bowen & Hinchey (2006). For more on the general application of safety engineering theory to AI, see Fox (1993); Yampolskiy & Fox (2013); Yampolskiy (2013).Another good point Fox makes is that normal AI safety engineering techniques rely on the design team's ability to predict all circumstances that might hold in the future: "…one might conclude that using a basket of safety methods (hazard analysis, formal specification and verification, rigorous empirical testing, fault tolerant design) will significantly decrease the likelihood of hazards and disasters. However, there is at least one weakness common to all these methods. They rely on the design team being able to make long-range predictions about all the… circumstances that may hold when the system is in use. This is unrealistic, if only because of the countless interactions that can occur… [and] the scope for unforseeable interactions is vast."See also this program at Bristol University.As an aside, I'll briefly remark that user interface confusion has contributed to many computer-related failures in the past. For example, Neumann (1994) reports on the case of Iran Air Flight 655, which was shot down U.S. forces due (partly) to the unclear user interface of the USS Vincennes' Aegis missile system. Changes to the interface were subsequently recommended. For other UI-related disasters, see Neumann's extensive page on Illustrative Risks to the Public in the Use of Computer Systems and Related Technology.The post Transparency in Safety-Critical Systems appeared first on Machine Intelligence Research Institute.

Transparency in Safety-Critical Systems


Holden Karnofsky on Transparent Research Analyses

Holden Karnofsky is the co-founder of GiveWell, which finds outstanding giving opportunities and publishes the full details of its analysis to help donors decide where to give. GiveWell tracked ~$9.6 million in donations made on the basis of its recommendations in 2012. It has historically sought proven, cost-effective, scalable giving opportunities, but its new initiative, GiveWell Labs, is more broadly researching the question of how to give as well as possible.

Luke Muehlhauser: GiveWell has gained respect for its high-quality analyses of some difficult-to-quantify phenomena: the impacts of particular philanthropic interventions. You've written about your methods for facing this challenge in several blog posts, for example (1) Futility of standardized metrics: an example, (2) In defense of the streetlight effect, (3) Why we can't take expected value estimates literally, (4) What it takes to evaluate impact, (5) Some considerations against more investment in cost-effectiveness estimates, (6) Maximizing cost-effectiveness via critical inquiry, (7) Some history behind our shifting approach to research, (8) Our principles for assessing research, (9) Surveying the research on a topic, (10) How we evaluate a study, and (11) Passive vs. rational vs. quantified.
In my first question I'd like to ask about one particular thing you've done to solve one particular problem with analyses of difficult-to-quantify phenomena. The problem I have in mind is that it's often difficult for readers to know how much they should trust a given analysis of a difficult-to-quantify phenomenon. In mathematics research it's often pretty straightforward for other mathematicians to tell what's good and what's not. But what about analyses that combine intuitions, expert opinion, multiple somewhat-conflicting scientific studies, general research in a variety of "soft" sciences, and so on? In such cases it can be difficult for readers to distinguish high-quality and low-quality analyses, and it can be hard for readers to tell whether the analysis is biased in particular ways.

One thing GiveWell has done to address this problem is to strive for an unusually high degree of transparency in its analyses. For example, your analyses often include:

Good summaries and bolding to make analyses more skimmable
External evaluations
Public conversation notes with credentialed experts
Detailed discussion of what you think and why
Lots of citations, direct quotes from studies, and footnotes
Archived copies of cited websites and papers

Do you agree with this interpretation of what you're doing? What are some other things you do to make it easier for readers to tell how they should update their views in response to your analyses?

Holden Karnofsky: Yes, that's a good characterization of what we're trying to do.
I'd say that our transparency efforts fall into two categories: "support" (making sure that all the relevant information about our decision-making processes is available to those who are seeking it) and "outreach" (proactively raising topics and views and inviting people to question us about them). We've found that "support" is often not enough: in order to get meaningful engagement on our work and help people become confident in it, we need to have high-level conversations in which we invite back-and-forth. Otherwise, people often feel that they don't have time to evaluate everything we've written, and don't engage with it.
Charity reviews, and most other pages on our main website (not the blog), tend to be in a "support" framework. The core of this framework is that for every statement we make, it should be clear what the basis of the statement is. If the basis of the statement is simply "our guess," that should be clear (so we often use language like "We'd guess that ___" rather than simply stating our beliefs without such a qualifier). If the basis of the statement is a more complex chain of reasoning that doesn't conceptually fit in the space, then such reasoning should be laid out elsewhere and linked to in the context of our statement. And if the basis of our statement is a document, website, or conversation, we use a footnote. Our conventions for footnotes go beyond what is common in academic papers: we generally aim to include the key quote (not just a citation) in a footnote, and we take steps (such as Webcite) to ensure that the original document and full context can be accessed by the reader even in the event that the original host of the source takes it down or changes it. We also try to structure charity reviews in logical ways and provide summary content at different levels of detail, so that it's always relatively quick to find out what we think on a given topic and determine how to drill down on a particular piece of it.
Most of the actual engagement we get with our views comes via "outreach" methods, particularly the blog and conference calls and in-person events (as well as one-on-one's with major donors to GiveWell and our top charities). The blog is probably the single most effective way of getting people to engage with, understand, and trust our research. Via the discussions these mediums create, we gain better understandings of where people's biggest questions and disagreements are, and we try to address these in further blog posts and FAQs. "Outreach" methods are organized around "What people want to know & what we want to tell them" rather than "What we think about charity X or intervention Y" and are often more informal, though they will link back to "support" content as appropriate.
In both frameworks, we rarely publish a piece of content without going over it and asking, "What are the key points here? Can they be summarized at the top, and/or bolded in the body?"

Luke: You identify GiveWell as part of the effective altruism movement, and you also write that "Effective altruism is unusual and controversial" — a contrarian position, we might say.
Robin Hanson notes that "On average, contrarian views are less accurate than standard views… Honest contrarians who expect reasonable outsiders to give their contrarian view more than normal credence should point to strong outside indicators that correlate enough with contrarians tending more to be right."
Do you agree with Hanson's analysis in that post? If so, then do you think GiveWell typically expects reasonable outsiders to give your contrarian views higher than normal credence, and are you able to point to "strong outside indicators that correlate enough with contrarians tending to be more right"?

Holden: I don't fully agree with that post. It implies that contrarian arguments need "strong outside indicators"; I think of a good argument as a combination of inside- and outside-view arguments, and enough strength in one area can make up for serious weakness in another. I'd frame it differently. I'd say that anyone who is trying to change minds through rational persuasion needs to think through whose minds they're trying to change and how much mental energy such people can reasonably be expected to put into evaluating their arguments (bearing in mind that reasonably strong low-detail arguments can create enough intrigue to get the audience to increase their mental energy investment and understand higher-detail arguments). What I agree with in Robin's post is that there is a non-trivial hurdle to overcome when espousing contrarian views, and (at least initially) a limited amount of mental energy one can expect the audience to invest in one's argument.
In our case, we've so far largely targeted people who already buy into effective altruism. So we don't deal with much of a "contrarian" dynamic for that specifically. Where I think we do have to "overcome the hurdles to contrarianism" is where we recommend charities that people haven't necessarily heard of, as opposed to charities that elite conventional wisdom recommends. In making the case for such charities, we do have to think about how to put a low-detail case up front rather than just refer people to our lengthy body of research as a whole. We try to follow the principle of "summarize our main points, and make it clear how to drill down on any particular one of them." That allows people to gain confidence in us via spot-check rather than by going through everything we've written (which I believe very few people have done).
In addition, tools like the blog can convince people over time that we're generally trustworthy and intelligent on the relevant issues, which can allow them to buy into our claims without evaluating all the details of them. (I perceive many people as having followed a similar path to supporting MIRI.)
But even with regard to recommending not-widely-known charities, we have a lot of wind at our backs by virtue of whom we're targeting. Much of our audience consists of people who already buy into effective altruism and already feel that the quality of dialogue around where to give is extraordinarily low. They also often come to us via referrals from trusted friends or media sources. So they arrive at our site very ready to believe that our claims are plausible.
As a side point, I think effective altruism falls somewhere on the spectrum between "contrarian view" and "unusual taste." My commitment to effective altruism is probably better characterized as "wanting/choosing to be an effective altruist" than as "believing that effective altruism is correct." I think that relieves some of the burden of having to "evaluate" effective altruism, though certainly not all of it.

Luke: Recently you've begun a new program within GiveWell called GiveWell Labs, which aims to investigate the effectiveness of causes that are even more difficult to analyze than e.g. global health interventions. In global health, you often get to learn from medical science, multiple randomized controlled trials for specific interventions, and so on. But it's much harder to investigate the cost effectiveness of interventions that aim to improve science, improve political or economic policy, reduce catastrophic risks, etc.
So I would imagine that developing GiveWell Labs has forced you to develop additional tools for analyzing complex phenomena, and for communicating your analyses to others. What are some things you've learned so far in the process of developing GiveWell Labs?

Holden: We're certainly developing new methods of analysis and evaluation. Our working framework for shallow investigations replaces "proven, cost-effective, scalable charities" with "important, tractable, non-crowded causes" in terms of what we're looking for. Much of our work so far has been more qualitative in nature, aiming to clarify and understand the basic landscape of causes rather than assess the extent to which approaches are "proven." And we've also been doing a fair amount of "immersion" – trying to learn broadly about a field without pre-choosing our set of critical questions and goals (for example, as I've started to investigate scientific research, I've read about half of a biology textbook and taken multiple chances to attend meetings of scientists). Relative to our traditional work, more of our learning so far on the Labs front comes from conversations as opposed to studies, and as a result our emphasis on (and volume of) conversation notes has increased dramatically.
With that said, it's still very early. I think we're fairly far from having concrete recommendations, and it's when we have concrete recommendations that it becomes much easier and more productive to work on engaging and convincing outsiders. We have certainly made attempts to put out what we've learned, using the same basic tools I mentioned before: "support" oriented pages (e.g., shallow investigation writeups) and "outreach" oriented communications (e.g., Labs-focused blog posts and Labs-focused in-person Q&A's).

Luke: I think GiveWell provides a helpful model of how to do valuable analytic research on difficult-to-quantify phenomena. Are there other groups doing a similar thing (but perhaps on other subjects) that you admire, or that you think are doing some important things right, or that you think are worth imitating in certain ways? You've praised the Cochrane Collaboration in the past… are there others?

Holden:

We're definitely fans of Cochrane. Their work is consistently "transparent" in the sense that one can read a review, get a sense of the big picture, know where to drill down on anything one wants to drill on, and ultimately – if one wants – answer just about any question about where their conclusions are coming form.
We've been impressed with the usefulness and, often, transparency of the Center for Global Development's work (more at this blog post).
I'm generally a fan of Nate Silver. While I wish he disclosed the full details of his models, I usually feel like I have a pretty good idea of what he's conceptually doing, without having to put much proactive effort into understanding it.
While there are many others whose work we use and/or enjoy, those are the only people that jump to mind in terms of "consistently providing transparent, convincing, reasonable analysis of difficult-to-analyze topics and/or topics relevant to GiveWell's work." I often see isolated cases of such work, such as particularly good academic papers.

Luke: Why do you think the sort of transparency you're providing is relatively rare, given that it seems to have some credibility benefits?

Holden: The sort of transparency we're providing takes a lot of work. Some of this is because our approach is relatively rare – for example, we've had to iterate a lot (and deal with some awkward situations) on being able to share notes from conversations. We've spoken to some organizations that seem potentially interested in increasing transparency, but with the processes and relationships they've built up, getting all the third parties they work with to buy in would be a huge project and struggle. It's easier for us because we've put this goal in our DNA from the beginning, but even for us the challenge of getting third parties to be comfortable with what we're doing can be significant.
Even if we weren't dealing with resistance from third parties, the time we put into writing up our thoughts would be a significant chunk of the total time we spend on research. Maybe half. So in some sense it seems like we could learn twice as much if we didn't feel the need to share our learning widely.
With that said, I think the benefits are significant as well, and I think they go beyond the credibility and influence boosts and feedback we get. Just the process of writing up and supporting our thinking often makes us think more clearly. We used to wait until our research felt "done" to write it up, but we found that the process of writing it up constantly caused us to rethink our conclusions and reopen investigations, as issues that had faded into the backgrounds of our minds re-emerged while we were trying to make our case. Now, we try to write things up "as" we research them, for exactly this reason. Writing things up for a general audience also means that our old material is easier for new employees (and long-running employees with not-so-great memories) to absorb, and so we lose less knowledge internally.
Even if we were allocating all of the "money moved" ourselves (rather than making giving recommendations to others) and even if no outsiders could ever see what we wrote, I'd still want to put a great deal of time – maybe almost as much as we do now – into creating clear, supported writeups and summaries.

Luke: Thanks, Holden!
The post Holden Karnofsky on Transparent Research Analyses appeared first on Machine Intelligence Research Institute.


2013 Summer Matching Challenge Completed!

Thanks to the generosity of dozens of donors, on August 15th we successfully completed the largest fundraiser in MIRI's history. All told, we raised $400,000, which will fund our research going forward.

This fundraiser came "right down to the wire." At 8:45pm Pacific time, with only a few hours left before the deadline, we announced on our Facebook page that we had only $555 more to raise to meet our goal. At 8:53pm, Benjamin Hoffman donated exactly $555, finishing the drive.
Our deepest thanks to all our supporters!
The post 2013 Summer Matching Challenge Completed! appeared first on Machine Intelligence Research Institute.


Luke at Quixey on Tuesday (Aug. 20th)

This coming Tuesday, MIRI's Executive Director Luke Muehlhauser will give a talk at Quixey titled Effective Altruism and the End of the World. If you're in or near the South Bay, you should come! Snacks will be provided.
Time: Tuesday, August 20th. Doors open at 7:30pm. Talk starts at 8pm. Q&A starts at 8:30pm.
Place: Quixey Headquarters, 278 Castro St., Mountain View, CA. (Google Maps)
Entrance: You cannot enter Quixey from Castro St. Instead, please enter through the back door, from the parking lot at the corner of Dana & Bryant.
The post Luke at Quixey on Tuesday (Aug. 20th) appeared first on Machine Intelligence Research Institute.

Luke at Quixey on Tuesday (Aug. 20th)


August Newsletter: New Research and Expert Interviews

Greetings from the Executive Director
Dear friends,
My personal thanks to everyone who has contributed to our ongoing fundraiser. We are 74% of the way to our goal!
I've been glad to hear from many of you that you're thrilled with the progress we've made in the past two years — progress both as an organization and as a research institute. I'm thrilled, too! And to see a snapshot of where MIRI is headed, take a look at the participant lineup for our upcoming December workshop. Some top-notch folks there, including John Baez.
We're also preparing for the anticipated media interest in James Barrat's forthcoming book, Our Final Invention: Artificial Intelligence and the End of the Human Era. The book reads like a detective novel, and discusses our research extensively. Our Final Invention will be released on October 1st by a division of St. Martin's Press, one of the largest publishers in the world.
If you're happy with the direction we're headed in, and you haven't contributed to our fundraiser yet, please donate now to show your support. Even small donations can make a difference. This newsletter is ~9,860 subscribers strong, and ~200 of you have contributed during the current fundraiser. If just 21% of the other 9,660 subscribers give $25 as soon as they finish reading this sentence, then we'll meet our goal will those funds alone!
Thank you,
Luke Muehlhauser
Executive Director

Summer Fundraiser Ends August 15th!
With only a few days left, we've raised 74% of our goal for our summer matching challenge. Many thanks to everyone who has contributed!
Donate on or before August 15th to double your donation and help us reach our goal of $200,000 raised ($400,000 with matching).
If we're able to reach our goal, then not only will we be able to continue to run research workshops and our other programs, but we might also be in a position later this year to hire our first new full-time mathematical researcher to work with Eliezer Yudkowsky on open problems in Friendly AI theory (e.g. the Löbian obstacle). We can't promise we'll decide that hiring a new FAI researcher is the optimal use of those funds at that time, but it is a serious option we're discussing internally. Our research workshops have been an excellent tool for evaluating potential hires.
Feel free to contact Luke Muehlhauser (<EMAIL>) directly for more details on how marginal funds will be used at MIRI, especially if you are considering a major gift ($5,000 or more).

Algorithmic Progress in Six Domains
MIRI has released a new technical report by Katja Grace: "Algorithmic Progress in Six Domains."
The report summarizes data on algorithmic progress – that is, better performance per fixed amount of computing hardware – in six domains:

SAT solvers,
Chess and Go programs,
Physics simulations,
Factoring,
Mixed integer programming, and
Some forms of machine learning.

MIRI's purpose for collecting these data was to shed light on the question of intelligence explosion microeconomics, though we suspect the report will be of broad interest within the software industry and computer science academia.

4 New Interviews; 2 New Analyses
Our blog was especially active this past month, with 4 new expert interviews and 2 new analyses.
New analyses:

What is AGI? A quick explanation of the concept of artificial general intelligence, and a selection of operational definitions that allow us to be even more specific about what we mean by "AGI."
AI Risk and the Security Mindset: "A recurring problem in much of the literature on 'machine ethics' or 'AGI ethics' or 'AGI safety' is that researchers and commenters often appear to be asking the question 'How will this solution work?' rather than 'How will this solution fail?'"

New expert interviews:

James Miller (economics, Smith College) on Unusual Incentives Facing AGI Companies
Roman Yampolskiy (computer science, U of Louisville) on AI Safety Engineering
Nick Beckstead (philosophy, Oxford) on the Importance of the Far Future
Benja Fallenstein (decision-making, Bristol U) on the Löbian Obstacle to Self-Modifying Systems

Burning Man Camp for Effective Altruists
The effective altruism movement (GiveWell, 80,000 Hours, etc.) is one of MIRI's intellectual communities. If you're going to Burning Man this year (Aug. 26 – Sep. 2) and would like to camp with many of MIRI's closest friends (including e.g. Anna Salamon of CFAR), then you may want to consider applying to the Burning Man theme camp for effective altruists, called Paradigm.
Paradigm has an excellent location on 6:30 and A, near Center Camp. Its organizer is Nevin Freeman (<EMAIL>). Paradigm will build a large dome called the Temple of Skeptical Consequentialism, which will host talks about effective altruism and rationality. Additional details, maps, and photos are here.
Spots are limited and many are already taken, so if you're interested, apply ASAP!

Job Openings at Giving What We Can and 80,000 Hours
Our friends at 80,000 Hours (80k) and Giving What We Can (GWWC) — two Oxford, UK organizations in the effective altruism movement — are hiring.
GWWC is hiring a Director of Communications, a Director of Community, and a Director of Research. Under "Why You Should Apply," they give these reasons:

Impact: A lot of charities will tell you that you can make a difference. Here we actually calculate that difference and we are driven by improving the measurable results of this organisation.
Inspiration: Offices don't come much more intellectually stimulating than ours. Based in offices shared with the Future of Humanity Institute at Oxford University, you'll be part of a team that dedicates itself to understanding how we can do the most good – and actually delivering on those ideas.
Personal development: All three positions offer fantastic personal development opportunities. We're looking for talent as well as experience – and you'll have opportunities to learn on the job, supported by a community dedicated to boosting their personal effectiveness.

80k is hiring a Careers Analyst, a Director of Fundraising, and a Finance Manager. See their (longer) pitch for why you should apply here.

Featured Volunteer – Bikramjeet Singh
Bikramjeet Singh is a 24 year old volunteer from India who first found out about MIRI's work when he was 16 – almost 8 years ago! He became acquainted with our deputy director Louie Helm 2 years later. After doing some work with promoting MIRI and volunteer searching, he was introduced to Michael Anissimov and became his personal media assistant. He remained in that position until February 2012, and joined the MIRIvolunteers.org system in December 2012. His motivation for volunteering for MIRI stems from his interest in existential risk reduction and his judgment that FAI is the best way to solve that problem. Bikramjeet's dream job is to be an AI researcher and a science fiction author.
Thanks for all your help Bikramjeet!

The post August Newsletter: New Research and Expert Interviews appeared first on Machine Intelligence Research Institute.