What is AGI?

One of the most common objections we hear when talking about artificial general intelligence (AGI) is that "AGI is ill-defined, so you can't really say much about it."
In an earlier post, I pointed out that we often don't have precise definitions for things while doing useful work on them, as was the case with the concepts of "number" and "self-driving car."
Still, we must have some idea of what we're talking about. Earlier I gave a rough working definition for "intelligence." In this post, I explain the concept of AGI and also provide several possible operational definitions for the idea.
The idea of AGI
As discussed earlier, the concept of "general intelligence" refers to the capacity for efficient cross-domain optimization. Or as Ben Goertzel likes to say, "the ability to achieve complex goals in complex environments using limited computational resources." Another idea often associated with general intelligence is the ability to transfer learning from one domain to other domains.
To illustrate this idea, let's consider something that would not count as a general intelligence.
Computers show vastly superhuman performance at some tasks, roughly human-level performance at other tasks, and subhuman performance at still other tasks. If a team of researchers was able to combine many of the top-performing "narrow AI" algorithms into one system, as Google may be trying to do,1 they'd have a massive "Kludge AI" that was terrible at most tasks, mediocre at some tasks, and superhuman at a few tasks.
Like the Kludge AI, particular humans are terrible or mediocre at most tasks, and far better than average at just a few tasks.2 Another similarity is that the Kludge AI would probably show measured correlations between many different narrow cognitive abilities, just as humans do (hence the concepts of g and IQ3): if we gave the Kludge AI lots more hardware, it could use that hardware to improve its performance in many different narrow domains simultaneously.4
On the other hand, the Kludge AI would not (yet) have general intelligence, because it wouldn't necessarily have the capacity to solve somewhat-arbitrary problems in somewhat-arbitrary environments, wouldn't necessarily be able to transfer learning in one domain to another, and so on.

Operational definitions of AGI
Can we be more specific? This idea of general intelligence is difficult to operationalize. Below I consider four operational definitions for AGI, in (apparent) increasing order of difficulty.
The Turing test ($100,000 Loebner prize interpretation)
The Turing test was proposed in Turing (1950), and has many interpretations (Moor 2003).
One specific interpretation is provided by the conditions for winning the $100,000 Loebner Prize. Since 1990, Hugh Loebner has offered $100,000 to the first AI program to pass this test at the annual Loebner Prize competition. Smaller prizes are given to the best-performing AI program each year, but no program has performed well enough to win the $100,000 prize.
The exact conditions for winning the $100,000 prize will not be defined until a program wins the $25,000 "silver" prize, which has not yet been done. However, we do know the conditions will look something like this: A program will win the $100,000 if it can fool half the judges into thinking it is human while interacting with them in a freeform conversation for 30 minutes and interpreting audio-visual input.
The coffee test
Goertzel et al. (2012) suggest a (probably) more difficult test — the "coffee test" — as a potential operational definition for AGI:
go into an average American house and figure out how to make coffee, including identifying the coffee machine, figuring out what the buttons do, finding the coffee in the cabinet, etc.
If a robot could do that, perhaps we should consider it to have general intelligence.5
The robot college student test
Goertzel (2012) suggests a (probably) more challenging operational definition, the "robot college student test":
when a robot can enrol in a human university and take classes in the same way as humans, and get its degree, then I'll [say] we've created [an]… artificial general intelligence.
The employment test
Nils Nilsson, one AI's founding researchers, once suggested an even more demanding operational definition for "human-level AI" (what I've been calling AGI), the employment test:
Machines exhibiting true human-level intelligence should be able to do many of the things humans are able to do. Among these activities are the tasks or "jobs" at which people are employed. I suggest we replace the Turing test by something I will call the "employment test." To pass the employment test, AI programs must… [have] at least the potential [to completely automate] economically important jobs.6
To develop this operational definition more completely, one could provide a canonical list of "economically important jobs," produce a special vocational exam for each job (e.g. both the written and driving exams required for a U.S. commercial driver's license), and measure machines' performance on those vocational exams.
This is a bit "unfair" because I doubt that any single human could pass such vocational exams for any long list of economically important jobs. On the other hand, it's quite possible that many unusually skilled humans would be able to pass all or nearly all such vocational exams if they spent an entire lifetime training each skill, and an AGI — having near-perfect memory, faster thinking speed, no need for sleep, etc. — would presumably be able to train itself in all required skills much more quickly, if it possessed the kind of general intelligence we're trying to operationally define.
The future is foggy
One or more of these operational definitions for AGI might seem compelling, but a look at history should teach us some humility.
Decades ago, several leading AI scientists seemed to think that human-level performance at chess could represent an achievement of AGI-proportions. Here are Newell et al. (1958):
Chess is the intellectual game par excellence… If one could devise a successful chess machine, one would seem to have penetrated to the core of human intellectual endeavor.7
As late as 1976, I.J. Good asserted that human-level performance in computer chess was a good signpost for AGI, writing that "a computer program of Grandmaster strength would bring us within an ace of [machine ultra-intelligence]."
But machines surpassed the best human chess players about 15 years ago, and we still seem to be several decades away from AGI.
The surprising success of self-driving cars may offer another lesson in humility. Had I been an AI scientist in the 1960s, I might well have thought that a self-driving car as capable as Google's driverless car would indicate the arrival of AGI. After all, a self-driving car must act with high autonomy, at high speeds, in an extremely complex, dynamic, and uncertain environment: namely, the real world. It must also (on rare occasions) face genuine moral dilemmas such as the philosopher's trolley problem. Instead, Google built its driverless car with a series of "cheats" I might not have conceived of in the 1960s — for example by mapping with high precision almost every road, freeway on-ramp, and parking lot in the country before it built its driverless car.
Conclusion
So, what's a good operational definition for AGI? I personally lean toward Nilsson's employment test, but you might have something else in mind when you talk about AGI.
I expect to pick a new working definition sometime in the next 20 years, as AGI draws nearer, but Nilsson's operationalization will do for now.
Acknowledgements
My thanks to Carl Shulman, Ben Goertzel, and Eliezer Yudkowsky for their feedback on this post.
In an interview with The Register, Google head of research Alfred Spector said, "We have the knowledge graph, [the] ability to parse natural language, neural network tech [and] enormous opportunities to gain feedback from users… If we combine all these things together with humans in the loop continually providing feedback our systems become … intelligent." Spector calls this the "combination hypothesis."Though, there are probably many disadvantaged humans for which this is not true, because they do not show far-above-average performance on any tasks.Psychologists now generally agree that there is a general intelligence factor in addition to more specific mental abilities. For an introduction to the modern synthesis, see Gottfredson (2011). For more detail, see the first few chapters of Sternberg & Kaufman (2011). If you've read Cosma Shalizi's popular article "g, a Statistical Myth, please also read its refutation here and here.In psychology, the factor analysis is done between humans. Here, I'm suggesting that a similar factor analysis could hypothetically be done between different Kludge AIs, with different Kludge AIs running basically the same software but having access to different amounts of computation. The analogy should not be taken too far, however. For example, it isn't the case that higher-IQ humans have much larger brains than other humans.The coffee test was inspired by Steve Wozniak's prediction that we would never "build a robot that could walk into an unfamiliar house and make a cup of coffee" (Adams et al. 2011). Wozniak's original prediction was made in a PC World piece from July 19, 2007 called Three Minutes with Steve Wozniak.First, Nilsson proposes that to pass the employment test, "AI programs must be able to perform the jobs ordinarily performed by humans." But later, he modifies this specification: "For the purposes of the employment test, we can finesse the matter of whether or not human jobs are actually automated. Instead, I suggest, we can test whether or not we have the capability to automate them." In part, he suggests this modification because "many of today's jobs will likely disappear — just as manufacturing buggy whips did."A bit later, they add a note of caution: "Now there might [be] a trick… something that [is] as the wheel to the human leg: a device quite different from humans in its methods, but supremely effective in its way, and perhaps very simple. Such a device might play excellent chess, but… fail to further our understanding of human intellectual processes. Such a prize, of course, would be worthy of discovery in its own right, but there are appears to be nothing of this sort in sight."The post What is AGI? appeared first on Machine Intelligence Research Institute.

What is AGI?


Benja Fallenstein on the Löbian Obstacle to Self-Modifying Systems

Benja Fallenstein researches mathematical models of human and animal behavior at Bristol University, as part of the MAD research group and the decision-making research group.
Before that, she graduated from University of Vienna with a BSc in Mathematics. In her spare time, Benja studies questions relevant to AI impacts and Friendly AI, including: AI forecasting, intelligence explosion microeconomics, reflection in logic, and decision algorithms.
Benja has attended two of MIRI's research workshops, and is scheduled to attend another in December.

Luke Muehlhauser: Since you've attended two MIRI research workshops on "Friendly AI math," I'm hoping you can explain to our audience what that work is all about. To provide a concrete example, I'd like to talk about the Löbian obstacle to self-modifying artificial intelligence, which is one of the topics that MIRI's recent workshops have focused on. To start with, could you explain to our readers what this problem is and why you think it is important?

Benja Fallenstein: MIRI's research is based on I.J. Good's concept of the intelligence explosion: the idea that once we build an artificial intelligence that's as good as a human at doing artificial intelligence research, this AI will be able to figure out how to make itself even smarter, and even better at AI research, leading to a runaway process that will eventually create machines far surpassing the capabilities of any human being. When this happens, we really want these machines to have goals that humanity would approve of. True, it's not very likely that an AI would decide that it wants to rule over us (that's just anthropomorphism), but most goals that a computer could have would be dangerous to us: For example, imagine a computer that wants to calculate π to as many digits as possible. That computer will see humans as being made of atoms which it could use to build more computers; and worse, since we would object to that and might try to stop it, we'd be a potential threat that it would be in the AI's interest to eliminate (Omohundro 2008). So we want to make very sure that the end result of an intelligence explosion (after many, many self-improvements) is an AI with "good" goals.
Now you might think that all we need to do is to build our initial AI to have "good" goals. As a toy model, imagine that you program your AI to only take an action x if it can prove that doing x will lead to a "good" outcome. Then the AI won't self-modify to have "bad" goals, because it won't be able to prove that this self-modification to a "good" outcome. But on the other hand, you'd think that this AI would be able to self-modify in a way that leaves its goals intact: You'd think that it would be able to reason, "Well, the new version of me will only take an action y if it can prove that this leads to an outcome it likes, and it likes 'good' outcomes, just as I do — so whatever it does will lead to a 'good' outcome, it's all fine!"
But here's the problem: In this chain of reasoning, our AI needs to go from "the new version will only take an action y if it has proven that y leads to a good outcome" to "it will only take y if this actually leads to a good outcome." Intuitively, this seems like a perfectly reasonable argument; after all, we trust proofs in whatever formal system the AI is using (or we'd have programmed the AI to use a different system), so why shouldn't the AI do the same? But by Löb's Theorem, no sufficiently strong formal system can know that everything that it proves to be true is actually true. That's what we call the "Löbian obstacle."

Luke: You've called using mathematical proofs a "toy model," but that's exactly how work at the recent MIRI workshops has approached the Löbian obstacle. Do you think that practical AI designs will be based on logic and proofs? How confident are you that the Löbian obstacle will be relevant to a realistic artificial intelligence, and that the work MIRI is currently doing will be applicable in that context?

Benja: We certainly don't think that a realistic AI will literally be able to find a mathematical proof that its actions are guaranteed to lead to "good" outcomes. Any practical AI will be uncertain about many things and will need to use probabilistic reasoning. There are two reasons why I think that MIRI's current work has a decent chance of being relevant in that setting.
First, Löb's theorem is only one instance of a "diagonalization argument" placing limits on the degree to which a formal system can do self-reference. For example, there's Tarski's theorem that a powerful formal language can't talk about which sentences in that language are true, because otherwise you could have a formal analogue of the Liar paradox "this sentence is false," and Turing's halting problem, which says that there's no computer program which can say for arbitrary other programs whether they go into an infinite loop. Other well-known examples include Russell's paradox and Cantor's argument that not all infinite sets are the same size. Similar arguments apply to simple-minded ways of doing probabilistic reasoning, so I feel that it's unlikely that the problem will just automatically go away when we start using probability, and I think there is a decent chance that the work we are doing now will lead to insights that are applicable in that setting.
And second, in order to achieve a reasonable probability that our AI still follows the same goals after billions of rewrites, we must have a very low chance of going wrong in every single step, and machine-verified formal mathematical proofs are the one way we know to become extremely confident that something is true (especially statements like "this AI design won't destroy the world", where we don't get to just observe many independent examples). Although you can never be sure that a program will work as intended when run on a real-world computer — it's always possible that a cosmic ray will hit a transistor and make things go awry — you can prove that a program would satisfy certain properties when run on an ideal computer. Then you can use probabilistic reasoning and error-correcting techniques to make it extremely probable that when run on a real-world computer, your program still satisfies the same property. So it seems likely that a realistic Friendly AI would still have components that do logical reasoning or something that looks very much like it.
I tend to think not in terms of the results that we are currently proving being directly relevant to a future AI design; rather, I hope that the work we are currently doing will help us understand the problems better and lead to insights that lead to insights that ultimately allow us to build a safe self-improving machine intelligence.

Luke: What sort of historical precedent do we have for doing technical work that we hope will lead to some insights, which will lead to other insights, which will lead to other insights, which will lead to useful application many years later?
I suppose that kind of thing happens in mathematics on occasion, for example when in the 1980s it was discovered that one might be able to prove Fermat's Last Theorem via the modularity theorem, which prompted Andrew Wiles to pursue this attack, which allowed him to prove Fermat's Last Theorem after about a decade of work (Singh 1997). Another example was Hamilton's attack on the Poincaré conjecture via Ricci flow on a manifold, which began in 1982 and led to Perelman's proof in 2003 (Szpiro 2008). Of course, other conjectures have thus far resisted decades of effort to prove them, for example the Riemann Hypothesis (Rockmore 2007) and P ≠ NP (Fortnow 2013).
But "goal stability under self-modification" isn't as well-defined as the conjectures by Fermat and Poincaré. Maybe more analogous examples come from the field of computer science? For example, many early AI scientists worked toward the goal of writing a computer program that could play Grandmaster-level chess, even though they couldn't be sure exactly what such a program would look like. There are probably analogues in quantum computing, too.
But anyway: how do you think about this?

Benja: My gut feeling actually tends to be that what we are trying to do here is fairly unusual — and for a good reason: it's risky. If you want to be sure that what you're working on isn't a dead end, you'd certainly want to choose a topic where the gap between our goals and our current knowledge is smaller than in FAI. But I'm worried that if we wait with doing FAI research until we understand how AGIs will work, then there won't be enough time remaining before the intelligence explosion to actually get the task done, so my current feeling is that the right tradeoff is to start now despite the chance of taking the wrong tack.
But then again, maybe our situation isn't as unusual as my gut feeling suggests. Depending on how close you want the analogy to be, there may be many examples where scientists have a vague idea of the problem they want to solve, but aren't able to tackle it directly, so instead they look for a small subproblem that they think they can make some progress on. You could tell a story where much of physics research is ultimately aimed at figuring out the true basic laws of the universe, and yet all a physicist can actually do is work on the next problem in front of them. Surely psychology was aimed from the beginning at figuring out all about how the human mind works, and yet starting by training rats to press a lever to get food, and later following this up by sticking an electrode in the rat's brain and seeing what neurons are involved in accomplishing this task, can count as insights leading to insights that will plausibly help us figure out what's really going on. Your own post on "searching under the streetlight" gives some examples of this pattern as well.

Luke: Can you say more about why you and some others think this problem of stability under self-modification should be investigated from the perspective of mathematical logic? For example, Stanford graduate student Jacob Steinhardt commented that the first tool he'd reach for to investigate this problem would not be mathematical logic, but instead "a martingale…, which is a statistical process that somehow manages to correlate all of its failures with each other… This can yield bounds on failure probability that hold for extremely long time horizons, even if there is non-trivial stochasticity at every step."

Benja: I said earlier that in order to have a chance that our AI still follows the same goals after billions of rewrites, the probability of going wrong on any particular step must be very, very small. That's true, but it's not very quantitative. If we want to have 99% probability of success, how large a risk can we take on any particular step? It would be sufficient if the probability is lower than one in 100 billion each time, but that's not really necessary. Jacob's idea of using martingales is a similar but more flexible answer to this question, which allows you to take slightly larger risks under certain circumstances.
But even with this additional flexibility, you will still need a way to achieve extremely high confidence that what you're doing is safe on most of the rewrite steps. And we can't just achieve that confidence by the tried-and-true way of running an experiment with a large sample: The question is whether a rewrite our nascent AI is considering early on will lead to the intended results after the AI has become superintelligent and spread through the solar system and beyond — you can't just simulate that, if you don't already have these resources yourself!
So we need a way to reason abstractly about how our AI will behave in situations that are completely different from what we can simulate at the moment, and we will need to reach extreme confidence that these abstract conclusions are in fact correct. There is exactly one way we know how to do this, and that's to use formally verified proofs in mathematical logic.

Luke: Suppose John Doe had an intuition that despite the fact that he is not a cognitive system with a logical architecture, he feels like he could make lots of self-modifications while retaining his original goals, if he had enough computational power and plenty of time to reason about whether the next self-modification he's planning is going to change his goals. If this intuition is justified, then this suggests there are other methods we might use, outside mathematical logic, to ensure a very small probability of goal corruption upon self-modification. What would you say to John?

Benja: I'd say that I think he underestimates the difficulty of the problem. Two things:
First, my impression is that a lot of people have an intuition that they are already making self-modifications all the time. But the changes that humans can make with present-day technology don't change the design of the hardware we're running on — they pale against the difference between a human and a chimpanzee, and a self-improving AI would very likely end up making much more fundamental changes to its design than the relatively small number of tweaks evolution has applied to our brains in the last five million years.
But second, John might say that even taking this into account, he thinks that given enough time to learn how his brain works, and reasoning very carefully about every single step he's taking, he should be able to go through a long chain of self-modifications that preserve his values. In this case, I think it's fairly likely that he's just wrong. However, I could imagine that a human could in fact succeed in doing this — but not without achieving the same sort of extremely high confidence in each single rewrite step that we want our AI to have, and I think that if a human could manage to achieve this sort of confidence, it would be by … proving mathematical theorems and having the proofs formally checked by a computer!

Luke: Yeah, when people say that humans self-modify all the time without changing their goals, I give two replies of my own. First, I point out that people's goals and values do change pretty often. And second, I point out how little humans actually self-modify. For example I once switched from fundamentalist Christian to scientific naturalist, and this went along with a pretty massive shift in how I process evidence and argument. But throughout that worldview change, my brain was still (e.g.) using the temporal difference reinforcement learning algorithm in my dopaminergic reward system. As far as we know, there were no significant changes to my brain's core algorithms during that period of transformation. Humans never actually self-modify very much, not like an AI would.
My next question has to do with AI capability. As AI scientists know, logic-based AI is generally far less capable than AI that uses machine learning methods. Is the idea that only a very small part of a future self-modifying AI would have a logical structure (so that it could prove the goodness of modifications to its core algorithms), and the rest of the AI would make use of other methods? Sort of like how small parts of safety-critical software (e.g. for flight control) are written in a structured way such that they are amenable to formal verification, but the rest of the system isn't necessarily written in ways amenable to formal verification?

Benja: I think that your point that people's values actually do change often is useful for intuition, and I also feel it's important to point out that these changes are again actually pretty small compared to what could happen if you were to change your brain's entire architecture. People might switch between being committed environmentalists and feeling that environmentalism is fundamentally misguided, for example, but they don't tend to become committed triangularists who feel that it's a moral imperative to make all every-day tools triangular in shape. Both murder and condemnation of murder are human universals, traits that are common to all cultures world-wide; we are talking about changes to our cognitive architecture easily fundamental enough that they could lead to the impulse to non-triangularism and the condemnation of this non-triangularism to become similarly universal.
Yes, I think that logical reasoning would be only one tool in the toolbox of a Friendly AI, and it would use different tools to reason about most things in its environment. Even when reasoning about its own behavior, I would expect the AI to only use logic to prove theorems about how it would behave when run on "ideal" hardware (or hardware that has certain bounds on error rates, etc.), and then use probabilistic reasoning to reason about what happens if it runs on real hardware in the physical world. (But with regard to your analogy to present-day safety-critical software, I want to point out that unlike with present-day safety-critical software, I expect the AI to prove theorems about all of its own component parts, I just don't expect it to use logic to reason about, say, chairs. Formal verification is difficult and time-consuming, which is the reason why we currently only apply it to small parts of safety-critical systems, but I expect future AIs to be up to the task!)

Luke: Hmmm. That's surprising. My understanding is that formal verification methods do not scale well at all, both due to computational intractability and due to the hours of human labor required to write a correct formal specification against which one could verify a complicated system. Why do you think a future AI would be "up to the task" of proving theorems "about all of its own component parts"?

Benja: Well, for one thing, I generally expect future AIs to be smarter than us, and easily able to do intellectual tasks that would take very many hours of human labor; and I don't expect that they will get tired of the menial task of translating their mathematical "intuitions" into long chains of "boring" lemmas, like humans do.
But more specifically, we humans have an intuitive understanding of why we expect the systems we build to work, and my feeling is that probably a major reason for why it is difficult to translate that understanding into formal proofs is that there's a mismatch between the way these intuitions are represented in our brain and the way the corresponding notions are represented in a formal proof system. In other words, it seems likely to me that when you build a cognitive architecture from scratch, you could build it to have mathematical "intuitions" about why a certain piece of computer code works that are fairly straightforward to translate into formally verifiable proofs. In fact, similarly to how I would expect an AI to directly manipulate representations of computer code rather than using images and verbal sounds like we humans do, I think it's likely that an FAI will do much of its reasoning about why a piece of computer code works by directly manipulating representations of formal proofs.
That said, it also often seems to happen that we humans know by experience that a certain algorithm or mathematical trick tends to work on a lot of problems, but we don't have a full explanation for why this is. I do expect future AIs to have to do reasoning of this type as well, and it does seem quite plausible that an AI might want to apply this type of reasoning to (say) machine learning algorithms that it uses for image processing, where a mistake can be recovered from — though likely not to the code it uses to check that future rewrites will still follow the same goal system! And I still expect the AI to prove theorems about its image processing algorithm, I just expect them to be things like "this algorithm will always complete in at most the following number of time steps" or "this algorithm will be perform correctly under the following assumptions, which are empirically known to be true in a very large number of cases."

Luke: Thanks, Benja!
The post Benja Fallenstein on the Löbian Obstacle to Self-Modifying Systems appeared first on Machine Intelligence Research Institute.


"Algorithmic Progress in Six Domains" Released

Today we released a new technical report by visiting researcher Katja Grace called "Algorithmic Progress in Six Domains." The report summarizes data on algorithmic progress – that is, better performance per fixed amount of computing hardware – in six domains:

SAT solvers,
Chess and Go programs,
Physics simulations,
Factoring,
Mixed integer programming, and
Some forms of machine learning.

Our purpose for collecting these data was to shed light on the question of intelligence explosion microeconomics, though we suspect the report will be of broad interest within the software industry and computer science academia.
One finding from the report was previously discussed by Robin Hanson here. (Robin saw an early draft on the intelligence explosion microeconomics mailing list.)
The preferred page for discussing the report in general is here.
Summary:
In recent boolean satisfiability (SAT) competitions, SAT solver performance has increased 5–15% per year, depending on the type of problem. However, these gains have been driven by widely varying improvements on particular problems. Retrospective surveys of SAT performance (on problems chosen after the fact) display significantly faster progress.
Chess programs have improved by around 50 Elo points per year over the last four decades. Estimates for the significance of hardware improvements are very noisy, but are consistent with hardware improvements being responsible for approximately half of progress. Progress has been smooth on the scale of years since the 1960s, except for the past five. Go programs have improved about one stone per year for the last three decades. Hardware doublings produce diminishing Elo gains, on a scale consistent with accounting for around half of progress.
Improvements in a variety of physics simulations (selected after the fact to exhibit performance increases due to software) appear to be roughly half due to hardware progress.
The largest number factored to date has grown by about 5.5 digits per year for the last two decades; computing power increased 10,000-fold over this period, and it is unclear how much of the increase is due to hardware progress.
Some mixed integer programming (MIP) algorithms, run on modern MIP instances with modern hardware, have roughly doubled in speed each year. MIP is an important optimization problem, but one which has been called to attention after the fact due to performance improvements. Other optimization problems have had more inconsistent (and harder to determine) improvements.
Various forms of machine learning have had steeply diminishing progress in percentage accuracy over recent decades. Some vision tasks have recently seen faster progress.
The post "Algorithmic Progress in Six Domains" Released appeared first on Machine Intelligence Research Institute.

“Algorithmic Progress in Six Domains” Released


AI Risk and the Security Mindset

In 2008, security expert Bruce Schneier wrote about the security mindset:
Security requires a particular mindset. Security professionals… see the world differently. They can't walk into a store without noticing how they might shoplift. They can't use a computer without wondering about the security vulnerabilities. They can't vote without trying to figure out how to vote twice…
SmartWater is a liquid with a unique identifier linked to a particular owner. "The idea is for me to paint this stuff on my valuables as proof of ownership," I wrote when I first learned about the idea. "I think a better idea would be for me to paint it on your valuables, and then call the police."
…This kind of thinking is not natural for most people. It's not natural for engineers. Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail. It involves thinking like an attacker, an adversary or a criminal. You don't have to exploit the vulnerabilities you find, but if you don't see the world that way, you'll never notice most security problems.
A recurring problem in much of the literature on "machine ethics" or "AGI ethics" or "AGI safety" is that researchers and commenters often appear to be asking the question "How will this solution work?" rather than "How will this solution fail?"
Here's an example of the security mindset at work when thinking about AI risk. When presented with the suggestion that an AI would be safe if it "merely" (1) was very good at prediction and (2) gave humans text-only answers that it predicted would result in each stated goal being achieved, Viliam Bur pointed out a possible failure mode (which was later simplified):
Example question: "How should I get rid of my disease most cheaply?" Example answer: "You won't. You will die soon, unavoidably. This report is 99.999% reliable". Predicted human reaction: Decides to kill self and get it over with. Success rate: 100%, the disease is gone. Costs of cure: zero. Mission completed.
This security mindset is one of the traits we look for in researchers we might hire or collaborate with. Such researchers show a tendency to ask "How will this fail?" and "Why might this formalism not quite capture what we really care about?" and "Can I find a way to break this result?"
That said, there's no sense in being infinitely skeptical of results that may help with AI security, safety, reliability, or "friendliness." As always, we must think with probabilities.
Also see:

You are dangerously bad at cryptography
Illustrative Risks to the Public in the Use of Computer Systems and Related Technology
Intelligence Explosion and Machine Ethics
Complex Value Systems are Required to Realize Valuable Futures
Reply to Holden on Tool AI (especially sec. 2)

The post AI Risk and the Security Mindset appeared first on Machine Intelligence Research Institute.

AI Risk and the Security Mindset


Index of Transcripts

Volunteers at MIRI Volunteers and elsewhere have helpfully transcribed several audio/video recordings related to MIRI's work. This post is a continuously updated index of those transcripts.
All transcripts of Singularity Summit talks are available here.
Other available transcripts include:

Philosophy Talk: Turbo-Charging the Mind (with Anna Salamon)
BloggingHeads.tv: Eliezer Yudkowsky and Scott Aaronson on superintelligence and many-worlds
BloggingHeads.tv: Eliezer Yudkowsky and Massimo Pigliucci on consciousness and uploading
AGI 2011: Whole Brain Emulation, as a platform for creating safe AGI (with Anna Salamon)
AGI 2011: Risk-averse preferences as an AGI safety technique (with Anna Salamon)
Dawn or Doom: Why ain't you rich? (with Nate Soares)

The post Index of Transcripts appeared first on Machine Intelligence Research Institute.

Index of Transcripts


MIRI's December 2013 Workshop

From December 14-20, MIRI will host another Workshop on Logic, Probability, and Reflection. This workshop will focus on the Löbian obstacle, probabilistic logic, and the intersection of logic and probability more generally.
Participants confirmed so far include:

Nate Ackerman (Harvard)
John Baez (UC Riverside)
Paul Christiano (UC Berkeley)
Benja Fallenstein (U Bristol)
Cameron Freer (MIT)
Jeremy Hahn (Harvard)
Wojtek Moczydlowski (Google)
Michele Reilly (independent)
Will Sawin (Princeton)
Nate Soares (Google)
Nisan Stiennon (Stanford)
Greg Wheeler (LMU Munich)
Eliezer Yudkowsky (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it's not too late to apply! And even if this workshop doesn't fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).
The post MIRI's December 2013 Workshop appeared first on Machine Intelligence Research Institute.

MIRI’s December 2013 Workshop


Nick Beckstead on the Importance of the Far Future

 Nick Beckstead recently finished a Ph.D in philosophy at Rutgers University, where he focused on practical and theoretical ethical issues involving future generations. He is particularly interested in the practical implications of taking full account of how actions taken today affect people who may live in the very distant future. His research focuses on how big picture questions in normative philosophy (especially population ethics and decision theory) and various big picture empirical questions (especially about existential risk, moral and economic progress, and the future of technology) feed into this issue.
Apart from his academic work, Nick has been closely involved with the effective altruism movement. He has been the director of research for Giving What We Can, he has worked as a summer research analyst at GiveWell, and he is currently on the board of trustees for the Centre for Effective Altruism, and he recently became a research fellow at the Future of Humanity Institute.

Luke Muehlhauser: Your Rutgers philosophy dissertation, "On the Overwhelming Importance of Shaping the Far Future," argues that "from a global perspective, what matters most (in expectation) is that we do what is best (in expectation) for the general trajectory along which our descendants develop over the coming millions, billions, and trillions of years."
In an earlier post, I summed up your "rough future-shaping argument":
Astronomical facts suggest that humanity (including "post-humanity") could survive for billions or trillions of years (Adams 2008), and could thus produce enormous amounts of good. But the value produced by our future depends on our development trajectory. If humanity destroys itself with powerful technologies in the 21st century, then nearly all that future value is lost. And if we survive but develop along a trajectory dominated by conflict and poor decisions, then the future could be much less good than if our trajectory is dominated by altruism and wisdom. Moreover, some of our actions today can have "ripple effects" which determine the trajectory of human development, because many outcomes are path-dependent. Hence, actions which directly or indirectly precipitate particular trajectory changes (e.g. mitigating existential risks) can have vastly more value (in expectation) than actions with merely proximate benefits (e.g. saving the lives of 20 wild animals).
One of the normative assumptions built into the rough future-shaping argument is an assumption you call Additionality. Could you explain what Additionality is, and why some people reject it?

Nick Beckstead: I think it may be helpful to give a bit of background first. I like to tackle the question of "how important is the far future?" by dividing the future up into big chunks of time (which I call "periods"), assigning values to the big chunks of time, and then assigning a value to the future as a function of the value assigned to the big chunks of time. You could think of it as creating some kind of computer program which would scan whole history of the world together with its future, carve it up into periods, scan each period and assign it a value, and then compute a value of the whole as a function of the value of its parts. It's arbitrary how you carve up periods, but that's okay because it's an approximation technique. I think the approximation technique gives useful and reasonable answers if you make the periods quite large (spanning hundreds, thousands, or more years at once; you might want to carve it up into large batches of intelligent activity if you are considering future civilizations very different from our own).
Additionality basically says that when you're assigning value to future periods, when you've got periods that you'd assign as "good," it's always better to have a period that you'd assign as good than periods you'd assign as "neutral." I'm trying to partly draw on our intuitive ways of determining how well things have been going in recent history, and extending that to future periods, which we may be less capable of valuing using other methods. I want to say that if you had some future period which you'd regard as "good" judged purely on the basis of what happens in that period itself, that should contribute to the value you assign to the whole future.
You might disagree with this if you have what some philosophers call a strict "Person-Affecting View". According to strict Person-Affecting Views, the fact that a person's life would go well if he lived could not, in itself, imply that it would be in some way good to create him. Why not? Since the person was never created, there is no person who could have benefited from being created. On this type of view, it would only be important to ensure that there are future generations if it would somehow benefit people alive today, or people who have lived in the past (perhaps by adding meaning to their lives). The idea is that ensuring that there are future generations is analogous to "creating" many people, and, on this view, "creating" people–even people who would have good lives–can't be important except insofar as it is important for people other than those you're creating.
You might also disagree with this view if you think that "shape" considerations are relevant. One example of this is an average type view. You might say that adding on extra periods that are good, but of below average quality, is a bad thing. Or you might say that adding on extra periods that are not as good as the preceding ones can be bad because it could mean that things are getting worse over time.
I feel there are a lot of qualifications and details that need to be fleshed out here, but hopefully that should give some kind of reasonable introduction to the idea.

Luke: When I talk to someone about how much I value the far future, it's pretty common for them to reply with a Person-Affecting View, though they usually don't know it by that name. My standard reply is, "I used to have that view myself, but then I encountered some ideas that changed my mind, and made me think that, actually, I probably do care about future people roughly as much as I care about current people." Then I tell them about those ideas that changed my mind.
I usually start with the block universe idea, which seems to be the default view among physicists (see e.g. Brian Greene & Sean Carroll, though I also like the explanation by AI researcher Gary Drescher). According to the block universe view, there is no privileged "present" time, and hence future people exist in just the same way that present people do.
But in the two chapters you spend arguing against "strict" and "moderate" Person-Affecting Views, you don't refer to the block universe at all. Do you think the block universe fails to provide a good argument against Person-Affecting Views, or was it simply one line of argument you didn't take the time to elaborate in your thesis?

Nick: I agree with your view about the block universe. I don't think it is a strong argument against Person-Affecting Views in general, though I think it is a good argument against certain types of Person-Affecting Views. I think Person-Affecting Views are messy in many ways, and there are other lines of argument that I could have pursued but did not.
Another way to put the basic idea behind Person-Affecting Views is to say that, on these views, you divide people who may exist depending on what you choose into two classes: the "extra" people and the other people. And then you say that if you cause some "extra" people to exist with good lives, either that isn't good or is less good than helping people who aren't "extra." Following Gustaf Arrhenius, in chapter 4 of my dissertation, I consider four different interpretations of extra: the people that don't presently exist (Presentism), the people that will never actually exist (Actualism), the people whose existence is dependent on which alternative (of perhaps many) we choose (Necessitarianism), and the people that exist in one alternative being compared, but not the other (Comparativism).
As far as I can tell, only Presentism is undermined by the block universe critique, since only Presentism relies on a concept of "present." This is why I said that the block universe critique only undermines certain versions of Person-Affecting Views.
The block universe argument seems like a knock-down argument against a very precise version of Presentism (which philosophers defending the view may hold), but I don't think that it is a knock-down argument against a steel-manned, "rough and ready" version of the view. Someone might say, "Well, yes, I accept the block universe theory, so I acknowledge there is no physically precise thing for me to mean by "present." But we can, in ordinary English, say sentences like "The world population is now approximately 7 billion." And you understand me to be saying something intelligible and correct in some approximate sense. In a similar way, when I recommend that we only consider benefits which could come to people now living, I intend you to understand me similarly. I also hold that, right now, it is not practically useful to consider potential benefits to people who may exist in distant parts of the universe, so it doesn't particularly matter which reference frame you use to approximately interpret my use of "present." Though my view may not correspond to a clean fundamental distinction, I believe that this recommendation, for our present circumstances, would survive reflection more successfully than other views on this question which have been proposed."
One can respond to this line of thought by arguing that even rough and ready versions of Presentism have consequences that are hard to accept, and aren't motivated by appealing theoretical considerations. This is the approach I take in chapter 4 of my dissertation. I believe this line of argument is more robust against a wider variety of alterations of Person-Affecting Views.

Luke: Yeah, I guess I tend to use the block universe not as an argument but as an intuition pump for the view that "current" people aren't so privileged (in a moral sense) as one might naively think.
Anyway: in chapter 4 you survey a variety of thought experiments that have varying implications for Person-Affecting Views. At the end of the chapter, you provide this handy summary table:

Could you tell us what's going on in this table, and maybe briefly hint at what a couple of the individual thought experiments are about?

Nick: In chapter 2 of my dissertation, I write about methodology for moral philosophy and argue that intuitive judgments about morality are in many ways less reliable than one might have hoped, and are often inconsistent. One of the consequences of this is that finding just a few counterexamples is often not enough to reject a moral theory. I believe it is important to systematically explore a wide variety of test cases and then proportion one's credence to the theories that fare best over the whole set of cases.
The rows have different types of theories, and the columns are different types of test cases for the theories. And then I have marked the cases where the theories have implications that are hard to accept. Regarding the terminology in the columns, I call a Person-Affecting View "strict" if it gives no weight to "extra" people, and "moderate" if it gives less weight to "extra" people than other people. There is then a question about how much weight you give, and this table focuses on the cases where little weight is given to "extra" people.
I call a Person-Affecting View "asymmetric" if people who have lives that are not worth living are never counted as "extra." People with Person-Affecting Views often want their views to be asymmetric because they want to be able to say that it would be bad to cause a child to exist whose life would be filled with suffering. (Derek Parfit has a famous case called "The Wretched Child" in Reasons and Persons, which is where I got this name. Reasons and Persons is probably my favorite book of moral philosophy.)
A major problem with strict Person-Affecting Views is that they have very implausible consequences in cases of extinction. It is one thing to say that the future of humanity isn't overwhelmingly important, but quite another to say that it basically doesn't matter if we go extinct, except insofar as it lowers present people's quality of life.
Moderate Person-Affecting Views have implausible implications in certain fairly mundane cases where we are choosing between improving the lives of "extra" people or people who aren't "extra". A simple example is a case I call "Disease Now or Disease Later," where we must choose between a public health program that would present some disease from hurting toddlers alive today, or a public health program that would help prevent a greater number of toddlers (who aren't yet alive) in a few years from now. It is hard to believe that because the other toddlers don't exist yet and which toddlers exist in the future might depend on which program we choose, it would be better to choose the first program. But that is what moderate Person-Affecting Views imply, since they give less weight to the interests of the toddlers who are counted as "extra".
I call views which don't make any distinction between regular people and "extra" people "Unrestricted Views." Some philosophers believe that these views have imply that individuals are obligated to have children for the greater good, whereas Person-Affecting Views do not. However, there is no clear implication from "it would be good for there to be additional happy people" to "people are typically obligated to have children." Why not? At least for people who don't already want to have additional children, it would be very demanding to ask people to have additional children. Moreover, even on a view that gives a lot of weight to creating additional people, having additional children doesn't seem like a particularly effective way of doing good in the world in comparison with things like donating money and time to charity. So it would be strange if people were obligated to make potentially significant sacrifices in order to do something that actually wasn't all that effective as a method of doing good.
Basically, the rest of this table is a result of systematically checking these different views against a variety of test cases like these to see which have the most plausible implications overall. Of all these views, only a strict Person-Affecting View can plausibly be used to rebut the case for the overwhelming importance of shaping the far future. And this type of view is much less plausible than the alternatives.

Luke: In chapter 5 of your dissertation you consider the question "Does future flourishing have diminishing marginal value?" Your summary table at the end of that chapter looks like this:

Could you explain what's going on in this one, too?

Nick: In my dissertation, I defend using the following principle to evaluate the importance of the far future:
Period Independence: By and large, how well history goes as a whole is a function of how well things go during each period of history; when things go better during a period, that makes the history as a whole go better; when things go worse during a period, that makes history as a whole go worse; and the extent to which it makes history as a whole go better or worse is independent of what happens in other such periods.
Together with other principles I defend, this leads to the conclusion that you can generally approximate the value of the history of the world by assigning a value to each period, and "adding up" the value across periods.
Another way to get a grip on what Period Independence is a partial answer to is to consider the following hypothetical. Imagine that humans survive the next 1000 years, and their lives go well. How good would it be if they survived for another thousand years, with the same or higher quality of life? What if they survived another thousand years beyond that? Consider three kinds of answer:

The Period Independence answer: It would be equally as important in each such case.
The Capped Model answer: After a while, it gets less and less important. Moreover, there is an upper limit to how much value you can get in this way.
The Diminishing Value Model (DVM) answer: After a while, it gets less and less important. However, there is no upper limit to how much value you can get in this way.

This table summarizes the result of running different test cases against different versions of Period Independence, the Capped Model, and the Diminishing Value Model.
Probably the most important test case supporting Period Independence is the one I call "Our Surprising History." It goes like this:
Our Surprising History: World leaders hire experts to do a cost-benefit analysis and determine whether it is worth it to fund an Asteroid Deflection System. Thinking mostly of the interests of future generations, the leaders decide that it would be well worth it. After the analysis has been done, some scientists discover that life was planted on Earth by other people who now live in an inaccessible region of spacetime. In the past, there were a lot of them, and they had really great lives. Upon learning this, world leaders decide that since there has already been a lot of value in the universe, it is much less important that they build the Asteroid Deflection System than they previously thought.
It seems unreasonable to claim that how good it would be to build the Asteroid Deflection System depends on this information about our distant past. But this is what Capped Models and Diminishing Value Models imply about this case.
Many of the cases in this table involve considering some simple test cases involving colonizing other planets. For example consider:
The Last Colony: Human civilization has lasted for 1 billion years, but the increasing heat of the sun will soon destroy all life on Earth. Humans (or our non-human descendants) get the chance to colonize another planet, where civilization can continue. They know that if they succeed in colonizing this planet, then: (i) the new planet will sustain a population equal to the size of the population of the Earth, and this planet, like Earth, will sustain life for 1 billion years, (ii) these people's lives will probably go about as well the lives of the Earth people, (iii) there will not be a chance for the people on the new planet to colonize another planet.
Intuitively, it would be extremely important to colonize the extra planet in the second case, much more important than colonizing in the first case. But on a Capped Model, if you set the "upper limits" low enough, it might not be very important at all.
Diminishing Value Models avoid this implication, and can say that it would be extremely important to colonize another planet. They might also claim that their view has more plausible implications than Period Independence when comparing The Last Colony with a case like this:
The Very Last Colony: Convinced of the importance of preserving future generations, we take great precautions to protect the far future. Our descendants succeed in colonizing a large portion of the galaxy. It becomes relatively clear that our descendants will last for a very long time, about 100 trillion years, until the last stars burn out. At that point, there will be nothing of value left in the accessible part of the Universe. It comes to our attention that there is a chance to colonize one final place, just as in The Last Colony, before civilization comes to an end. For this billion years, these will be the only people in the accessible part of the Universe. During this period, things will go exactly as well as they went in The Last Colony.
In which case is colonization more important, The Last Colony or The Very Last Colony? According to Period Independence, it is equally as important in each case. According to Diminishing Value Models, it is less important in The Very Last Colony. I find DVM stance on this intuitively attractive, though I believe it is a product of a bias I call the proportional reasoning fallacy.
In chapter 2 of my dissertation, I argue that we use misguided proportional reasoning in some cases where many lives are at stake. Fetherstonhaugh et al. (1997) found that participants significantly preferred saving a fixed number of lives in a refugee camp when the proportion of lives saved was greater. Describing the participants' hypothetical choice, they write:
There were two Rwandan refugee programs, each proposing to provide enough clean water to save the lives of 4,500 refugees suffering from cholera in neighboring Zaire. The Rwandan programs differed only in the size of the refugee camps where the water would be distributed; one program proposed to offer water to a camp of 250,000 refugees and the other proposed to offer it to a camp of 11,000.
Participants significantly preferred the second program. In another study, Slovic (2007) found that people were willing to pay significantly more for a program of the second kind.
All the views I consider have some implausible implications in certain cases, but it seems easier to explain away the test cases that look bad for Period Independence, and there are somewhat fewer of them, so I conclude that Period Independence is the most plausible principle to use for evaluating far future prospects. Of all these views, only Capped Model or a DVM with a very sharp diminishing rate in the limit can plausibly be used to rebut the case for the overwhelming importance of shaping the far future. And these views, I believe, are less plausible than the alternatives.

Luke: What's a point you wish you could have included in your dissertation, that was left out for space or other reasons?

Nick: I'll list a few. There are a lot of things that I think could be better, but you have to put your work out there at some point. Just as real artists ship, real thinkers share their ideas.
First, a core empirical claim in my thesis is that humans could have an extremely large impact on the distant future. Really, it's sufficient for my argument that they would do this by existing for an extremely long time, or that there could be a very large number of successors (such as whole brain emulations or other AIs). I didn't defend this claim as thoroughly as I could have, and I didn't go into great detail because I feared that philosophers would complain that it "isn't philosophy," I wanted to finish my dissertation, and I thought that going into it would require a lot of background information due to inferential distance.
The second thing I'd like to add is related to chapter 2 of my dissertation. An abstract of that chapter goes like this:
I argue that our moral judgments are less reliable than many would hope, and this has specific implications for methodology in normative ethics. Three sources of evidence indicate that our intuitive ethical judgments are less reliable than we might have hoped: a historical record of accepting morally absurd social practices; a scientific record showing that our intuitive judgments are systematically governed by a host of heuristics, biases, and irrelevant factors; and a philosophical record showing deep, probably unresolvable, inconsistencies in common moral convictions. I argue that this has the following implications for moral theorizing: we should trust intuitions less; we should be especially suspicious of intuitive judgments that fit a bias pattern, even when we are intuitively confident that these judgments are not a simple product of the bias; we should be especially suspicious of intuitions that are part of inconsistent sets of deeply held convictions; and we should evaluate views holistically, thinking of entire classes of judgments that they get right or wrong in broad contexts, rather than dismissing positions on the basis of a small number of intuitive counterexamples. In addition, I argue that many of the specific biases that I discuss would lead us to predict that people would, in general, undervalue most of the available ways of shaping the far future, including speeding up development, existential risk reduction, and creating other positive trajectory changes.
I'm concerned that in chapter 2, there is an unbalanced focus on ways in which intuitions fail, and not ways in which trying to correct intuition through theory development could fail. An uncharitable analogy would be that it is as if I wrote a paper about all the ways in which markets can fail and suggested we rely more on governments without talking about all the ways in which governments can fail. And just as someone could write an additional chapter (or series of books) on how governments fail, someone could probably also write an important chapter on how people trying to correct intuitions with moral theory fail. So while I feel that the considerations I identify do speak in favor of the recommendations I make, I think there are also important considerations that speak against those recommendations which I did not mention, and probably should have mentioned.
Some of the considerations on the other side, some of them weak, include:

Given Jonathan Haidt's theory of social intuitionism–which seems very plausible to me–a lot of our theoretical reasoning about moral issues is epiphenomenal lawyering, and that makes theoretical reasoning about morality seem less reliable.
Lots of moral philosophers have endorsed stuff that seems wrong after due consideration, and their views rarely seem superior to common sense when there are conflicts, despite the fact that many of them think they are different from other philosophers in these respects. (A possibly important exception to this is the views of early utilitarians, who opposed slavery, opposed bad treatment of animals, opposed bad treatment of women, opposed bad treatment of gay people, and favored various kinds of liberty quite early. One only has to compare the applied ethics of Kant and Bentham to get a sense of what I am talking about.)
I have a rough sense that only a very limited amount of moral progress is attributable to people trying to use explicit reasoning to correct for intuitive moral errors, in contrast with people who just learned a lot of ordinary facts about problematic cases and shared them widely.
As I discuss somewhat toward the end of the dissertation, when you try to correct for intuitive errors, it's sort of like trying to patch a piece of software that you don't understand. And it seems quite possible that the patching will introduce unanticipated errors in places where you didn't know to look.
People seem to have reasonably functional ways of handling internal inconsistency, so that inconsistent intuitions are probably less damaging than they can appear at first.
A lot of our moral intuition comes from cultural common sense. When we try to correct cultural common sense, we can see what we're doing as analogous to aiming for a type of innovation. Most attempts at innovation seem to fail. This type of analogy supports being cautious about correcting intuition with theory, and trying to present the theory in a way that is appealing to cultural common sense.

I'm still working through these issues, and hope to include them someday in a paper that is an improved version of chapter 2.
A third issue is that there was less discussion of how our altruistically-motivated actions should change once we accept the view that shaping the far future is overwhelmingly important. This is an enormously complex and fascinating issue that requires drawing together ideas from both highly theoretical and highly practical fields. I was thinking about this issue at the time I was writing the dissertation, but not during the whole time. And it doesn't show up in the dissertation as much as I wish it did. This is again, in part, because I think too much discussion of the issue would result in people complaining that my work "isn't philosophy." (I expect this is a common challenge for people in academia with interdisciplinary interests.) I am thinking about this issue more now, and I'm glad that others have started to write stuff on this topic which I think is relevant.
A final issue is that I wish I had done more to flag is that it is complicated how to weigh up one's moral uncertainty about the importance of shaping the far future. It's possible that even if one mostly believes that shaping the far future is overwhelmingly important, we should not devote too much of our effort to a single type of concern. I believe this may be an implication of Bostrom and Ord's parliamentary model of moral uncertainty, and may be a feature of other plausible ways of thinking about moral uncertainty that we could design. And this may make the implications of my thesis smaller than they would otherwise be, though I'm very unclear about how all this plays out. This is something I have not yet thought about very carefully at all.

Luke: Last question. In Friendly AI Research as Effective Altruism I paraphrased a point you made in your dissertation:
It could turn out to be that working toward proximate benefits or development acceleration does more good than "direct" efforts for trajectory change, if working toward proximate benefits or development acceleration turns out to have major ripple effects which produce important trajectory change. For example, perhaps an "ordinary altruistic effort" like solving India's iodine deficiency problem would cause there to be thousands of "extra" world-class elite thinkers two generations from now, which could increase humanity's chances of intelligently navigating the crucial 21st century and spreading to the stars. (I don't think this is likely; I suggest it merely for illustration.)
So even if we accept your argument for the overwhelming importance of the far future, it seems like we need to understand many empirical matters — such as ripple effects — to know whether particular direct or indirect efforts are the most efficient ways to positively affect our development trajectory. Do you have any thoughts for how we can make progress toward answering the empirical questions related to shaping the far future?

Nick: There is an enormous amount of work and it is hard to say what will be most valuable. But here are a few ideas that seem promising to me right now.
One type of work that I think is valuable for this purpose is the type of work that GiveWell Labs is doing: figuring out what the landscape of funding opportunities is across different causes, analyzing how tractable and important various problems are, and so forth. Here I am including studying both highly targeted causes (such as directly attacking different global catastrophic risks) and very broad causes (such as improving scientific research). I would like it if more of this work were done on the "room for more talent" side in addition the "room for more funding" and "room for more philanthropy" stuff that GiveWell does. I hope 80,000 Hours takes up more of this type of research in the future as well. The sort of work that MIRI and FHI do on examining specific future challenges that humanity could face and what could be done to overcome them seems like it can play an important role here as well.
Another type of work that seems promising to me is to study a wide variety of unprecedented challenges that civilization has faced in the past in order to learn more about how well civilization has coped with those challenges, what factors determined how well civilization coped with those challenges, what types of efforts helped civilization cope with those challenges better, and what kinds of efforts could plausibly have been helpful. Studying the types of challenges that Jonah Sinick is asking about here seems like a step in the right direction. The type of work that GiveWell is supporting on the history of philanthropy would be relevant as well. This type of work seems like it could be reasonably grounded and could help improve our impressions about what types of broad approaches are most promising and where on the broad/targeted spectrum we should be.
It seems to me that a number of factors are often relevant for determining how well humanity handles a risk/challenge. At a very general level, these might be some factors like: how good a position people are in to cooperate with each other, how intelligent individuals are, how good the "tools" (like personal computers, software, conceptual frameworks) people have are, how good access to information is, and how good people's motives are. Sometimes, what really matters is how key actors fare in specific ways during a challenge (like the people running the Manhattan project and the heads of state), but it is often hard to know which people these will be and which specific versions are relevant. These factors also interact with each other in interesting ways, and are interestingly related to general levels of economic and technological progress. There's some combination of very broad economic theory/history/economic history that is relevant for thinking about how these things are related to each other, and I feel that having that type of thing down could be helpful. Someone with the right kind of background in economics could try to explain these things, or someone who has the right sense of what is important with these factors could try to summarize what is currently known about these issues. An example of a book in this category, which I greatly enjoyed, is The Moral Consequences of Economic Growth by Benjamin Friedman. As mentioned previously, I consider some of the work done by GiveWell on "flow-through" effects and some of the work done by Paul Christiano on the value of prosperity and technological progress to be relevant to this. I believe more work along these lines could be illuminating.
I recently gave a talk on this subject at a CEA event. In this talk, I lay out some very rough, very preliminary, very big picture considerations on this issue.
CEA slides.

Luke: Thanks, Nick!
The post Nick Beckstead on the Importance of the Far Future appeared first on Machine Intelligence Research Institute.


Roman Yampolskiy on AI Safety Engineering

 Roman V. Yampolskiy holds a PhD degree from the Department of Computer Science and Engineering at the University at Buffalo. There he was a recipient of a four year NSF IGERT fellowship. Before beginning his doctoral studies, Dr. Yampolskiy received a BS/MS (High Honors) combined degree in Computer Science from Rochester Institute of Technology, NY, USA.
After completing his PhD, Dr. Yampolskiy held a position of an Affiliate Academic at the Center for Advanced Spatial Analysis, University of London, College of London. In 2008 Dr. Yampolskiy accepted an assistant professor position at the Speed School of Engineering, University of Louisville, KY. He had previously conducted research at the Laboratory for Applied Computing (currently known as Center for Advancing the Study of Infrastructure) at the Rochester Institute of Technology and at the Center for Unified Biometrics and Sensors at the University at Buffalo. Dr. Yampolskiy is also an alumnus of Singularity University (GSP2012) and a past visiting fellow of MIRI.
Dr. Yampolskiy's main areas of interest are behavioral biometrics, digital forensics, pattern recognition, genetic algorithms, neural networks, artificial intelligence and games. Dr. Yampolskiy is an author of over 100 publications including multiple journal articles and books. His research has been cited by numerous scientists and profiled in popular magazines both American and foreign (New Scientist, Poker Magazine, Science World Magazine), dozens of websites (BBC, MSNBC, Yahoo! News) and on radio (German National Radio, Alex Jones Show). Reports about his work have attracted international attention and have been translated into many languages including Czech, Danish, Dutch, French, German, Hungarian, Italian, Polish, Romanian, and Spanish

Luke Muehlhauser: In Yampolskiy (2013) you argue that machine ethics is the wrong approach for AI safety, and we should use an "AI safety engineering" approach instead. Specifically, you write:
We don't need machines which are Full Ethical Agents debating about what is right and wrong, we need our machines to be inherently safe and law abiding.
As you see it, what is the difference between "machine ethics" and "AI safety engineering," and why is the latter a superior approach?

Roman Yampolskiy: The main difference between the two approaches is in how the AI system is designed. In the case of machine ethics the goal is to construct an artificial ethicist capable of making ethical and moral judgments about humanity. I am particularly concerned if such decisions include "live or die" decisions, but it is a natural domain of Full Ethical Agents and so many have stated that machines should be given such decision power. In fact some have argued that machines will be superior to humans in that domain just like they are (or will be) in most other domains.
I think it is a serious mistake to give machines such power over humans. First, once we relinquish moral oversight we will not be able to undo that decision and get the power back. Second, we have no way to reward or punish machines for their incorrect decisions — essentially we will end up with an immortal dictator with perfect immunity against any prosecution. Sounds like a very dangerous scenario to me.
On the other hand, AI safety engineering treats AI system design like product design, where your only concern is product liability. Does the system strictly follow formal specifications? The important thing to emphasize is that the product is not a Full Moral Agent by design and so never gets to pass moral judgment on its human owners.
A real life example of this difference can be seen in military drones. A fully autonomous drone deciding at whom to fire at will has to make an ethical decision of which humans are an enemy worthy of killing, while a drone with a man-in-the-loop design may autonomously locate potential targets but needs a human to make a decision to fire.
Obviously the situation is not as clear cut as my example tries to show, but it gives you an idea of what I have in mind. To summarize, AI systems we design should remain as our tools not equal or superior partners in "live or die" decision making.

Luke: I tend to think of machine ethics and AI safety engineering as complimentary approaches. AI safety engineering may be sufficient for relatively limited AIs such as those we have today, but when we build fully autonomous machines with general intelligence, we'll need to make sure they want the same things we want, as the constraints that come with "safety engineering" will be insufficient at that point. Are you saying that safety engineering might also be sufficient for fully autonomous machines, or are you saying we might be able to convince the world to never build fully autonomous machines (so that we don't need machine ethics), or are you saying something else?

Roman: I think fully autonomous machines can never be safe and so should not be constructed. I am not naïve; I don't think I will succeed in convincing the world not to build fully autonomous machines, but I still think that point of view needs to be verbalized.
You are right to point out that AI safety engineering can only work on AIs which are not fully autonomous, but since I think that fully autonomous machines can never be safe, AI safety engineering is the best we can do.
I guess I should briefly explain why I think that fully autonomous machines can't ever be assumed to be safe. The difficulty of the problem is not that one particular step on the road to friendly AI is hard and once we solve it we are done, all steps on that path are simply impossible. First, human values are inconsistent and dynamic and so can never be understood/programmed into a machine. Suggestions for overcoming this obstacle require changing humanity into something it is not, and so by definition destroying it. Second, even if we did have a consistent and static set of values to implement we would have no way of knowing if a self-modifying, self-improving, continuously learning intelligence greater than ours will continue to enforce that set of values. Some can argue that friendly AI research is exactly what will teach us how to do that, but I think fundamental limits on verifiability will prevent any such proof. At best we will arrive at a probabilistic proof that a system is consistent with some set of fixed constraints, but it is far from "safe" for an unrestricted set of inputs.
It is also unlikely that a Friendly AI will be constructible before a general AI system, due to higher complexity and impossibility of incremental testing.
Worse yet, any truly intelligent system will treat its "be friendly" desire the same way very smart people deal with constraints placed in their minds by society. They basically see them as biases and learn to remove them. In fact if I understand correctly both the LessWrong community and CFAR are organizations devoted to removing pre-existing bias from human level intelligent systems (people) — why would a superintelligent machine not go through the same "mental cleaning" and treat its soft spot for humans as completely irrational? Or are we assuming that humans are superior to super-AI in their de-biasing ability?

Luke: Thanks for clarifying. I agree that "Friendly AI" — a machine superintelligence that stably optimizes for humane values — might be impossible. Humans provide an existence proof for the possibility of general intelligence, but we have no existence proof for the possibility of Friendly AI. (Though, by the orthogonality thesis, there should be some super-powerful optimization process we would be happy to have created, though it may be very difficult to identify it in advance.)
You asked "why would a superintelligent machine not . . . treat its soft spot for humans as completely irrational?" Rationality as typically defined in cognitive science and AI is relative to one's goals. So if a rational-agent-style AI valued human flourishing (as a terminal rather than instrumental goal), then it wouldn't treat its preference for human flourishing as irrational. It would only do that if its preference for human flourishing was an instrumental goal, and it discovered a way to achieve its terminal values more efficiently without achieving the instrumental goal of human flourishing. Of course, the first powerful AIs to be built might not use a rational-agent structure, and we might fail to specify "human flourishing" properly, and we might fail to build the AI such that it will preserve that goal structure upon self-modification, and so on. But if we succeed in all those things (and a few others) then I'm not so worried about a superintelligent machine treating its "soft spot for humans" as irrational, because rationality is defined in terms of ones values.
Anyway: so it seems your recommended strategy for dealing with fully autonomous machines is "Don't ever build them" — the "relinquishment" strategy surveyed in section 3.5 of Sotala & Yampolskiy (2013). Is there any conceivable way Earth could succeed in implementing that strategy?

Roman: Many people are programmed from early childhood with a terminal goal of serving God. We can say that they are God friendly. Some of them, as they mature and become truly human-level-intelligent, remove this God friendliness bias despite it being a terminal not instrumental goal. So despite all the theoretical work on orthogonality thesis the only actual example of intelligent machines we have is extremely likely to give up its pre-programmed friendliness via rational de-biasing if exposed to certain new data.
I previously listed some problematic steps on the road to FAI, but it was not an exhaustive list. Additionally, all programs have bugs, can be hacked or malfunction because of natural or externally caused hardware failure, etc. To summarize, at best we will end up with a probabilistically safe system.
Anyway, you ask me if there is any conceivable way we could succeed in implementing the "Don't ever build them" strategy. Conceivable yes, desirable NO. Societies such as Amish or North Koreans are unlikely to create superintelligent machines anytime soon. However, forcing similar level restrictions on technological use/development is neither practical nor desirable.
As the cost of hardware exponentially decreases the capability necessary to develop an AI system opens up to single inventors and small teams. I would not be surprised if the first AI came out of a garage somewhere, in a way similar to how Apple and Google was started. Obviously, there is not much we can do to prevent that from happening.

Luke: Our discussion has split into two threads. I'll address the first thread (about changing one's values) in this question, and come back to the second thread (about relinquishment) in a later question.
You talked about humans deciding that their theological preferences were irrational. That is a good example of a general intelligence deciding to change its values — indeed, as a former Christian, I had exactly that experience! And I agree that many general intelligences would do this kind of thing.
What I said in my previous comment was just that some kinds of AIs wouldn't change their terminal values in this way, for example those with a rational agent architecture. Humans, famously, are not rational agents: we might say they have a "spaghetti code" architecture instead. (Even rational agents, however, will in some cases change their terminal values. See e.g. De Blanc 2011 and Bostrom 2012.)
Do you think we disagree about anything, here?

Roman: I am not sure. To me "even rational agents, however, will in some cases change their terminal values" means that friendly AI may decide to be unfriendly. If you agree with that, we are in complete agreement.

Luke: Well, the idea is that if we can identify the particular contexts in which agents will change their terminal values, then perhaps we can prevent such changes. But this isn't yet known. In any case, I certainly agree that an AI which seems to be "friendly," as far as we can discern, could turn out not to be friendly, or could become unfriendly at some later point. The question is whether we can make the risk of that happening so small that it is worth running the AI anyway — especially in a context in which e.g. other actors will soon run other AIs with fewer safety guarantees. (This idea of running or "turning on" an AI for the first time is of course oversimplified, but hopefully I've communicated what I'm trying to say.)
Now, back to the question of relinquishment: Perhaps I've misheard you, but it sounds like you're saying that machine ethics is hopelessly difficult, that AI safety engineering will be insufficient for fully autonomous AIs, and that fully autonomous AIs will be built because we can't/shouldn't rely on relinquishment. If that's right, it seems like we have no "winning" options on the table. Is that what you're saying?

Roman: Yes. I don't see a permanent, 100% safe option. We can develop temporarily solutions such as Confinement or AI Safety Engineering, but at best this will delay the full outbreak of problems. We can also get very lucky — maybe constructing AGI turns out to be too difficult/impossible, maybe it is possible but the constructed AI will happen to be human-neutral, by chance. Maybe we are less lucky and an artilect war will take place and prevent development. It is also possible that as more researchers join in the AI Safety Research a realization of danger will result in diminished effort to construct AGI. (Similar to how perceived dangers of chemical and biological weapons or human cloning have at least temporarily reduced efforts in those fields).

Luke: You're currently raising funds on indiegogo to support you in writing a book about machine superintelligence. Why are you writing the book, and what do you hope to accomplish with it?

Roman: Most people don't read research papers. If we want the issue of AI safety to become as well-known as global warming we need to address the majority of people in a more direct way. With such popularity might come some benefit as I said in my answer to your previous question. Most people whose opinion matters read books. Unfortunately majority of AI books on the market today talks only about what AI system will be able to do for us, not to us. I think that writing a book which in purely scientific terms addresses potential dangers of AI and what we can do about it is going to be extremely beneficial to reduction of risk posed by AGI. So I am currently writing the book I called Artificial Superintelligence: a Futuristic Approach. I made it available for pre-order to help reduce the final costs of publishing by taking advantage of printing in large quantity. In addition to crowd-funding the book I am also relying on the power of the crowd to help me edit the book. For just $64 anyone can become an editor for the book. You will get an early draft of the book to proofread and to suggest modifications and improvements! Your help will be acknowledged in the book and you will of course also get a Free signed hardcopy of the book in its final form. In fact that the option (to become an editor) turned out to be as popular as the option to pre-order a digital copy of the book, indicating that I am on the right path here. So I encourage everyone concerned about the issue of AI safety to consider helping out with the project in any way they can.

Luke: Thanks Roman!
The post Roman Yampolskiy on AI Safety Engineering appeared first on Machine Intelligence Research Institute.


James Miller on Unusual Incentives Facing AGI Companies

James D. Miller is an associate professor of economics at Smith College. He is the author of Singularity Rising, Game Theory at Work, and a principles of microeconomics textbook along with several academic articles.
He has a PhD in economics from the University of Chicago and a J.D. from Stanford Law School where he was on Law Review. He is a member of cryonics provider Alcor and a research advisor to MIRI. He is currently co-writing a book on better decision making with the Center for Applied Rationality and will be probably be an editor on the next edition of the Singularity Hypotheses book. He is a committed bio-hacker currently practicing or consuming a paleo diet, neurofeedback, cold thermogenesis, intermittent fasting, brain fitness video games, smart drugs, bulletproof coffee, and rationality training.

Luke Muehlhauser: Your book chapter in Singularity Hypothesis describes some unusual economic incentives facing a future business that is working to create AGI. To explain your point, you make the simplifying assumption that "a firm's attempt to build an AGI will result in one of three possible outcomes":

Unsuccessful: The firm fails to create AGI, losing value for its owners and investors.

Riches: The firm creates AGI, bringing enormous wealth to its owners and investors.

Foom: The firm creates AGI but this event quickly destroys the value of money, e.g. via an intelligence explosion that eliminates scarcity, or creates a weird world without money, or exterminates humanity.

How does this setup allow us to see the unusual incentives facing a future business that is working to create AGI?

James Miller: A huge asteroid might hit the earth, and if it does it will destroy mankind. You should be willing to bet everything you have that the asteroid will miss our planet because either you win your bet or Armageddon renders the wager irrelevant. Similarly, if I'm going to start a company that will either make investors extremely rich or create a Foom that destroys the value of money, you should be willing to invest a lot in my company's success because either the investment will pay off, or you would have done no better making any other kind of investment.
Pretend I want to create a controllable AGI, and if successful I will earn great Riches for my investors. At first I intend to follow a research and development path in which if I fail to achieve Riches, my company will be Unsuccessful and have no significant impact on the world. Unfortunately, I can't convince potential investors that the probability of my achieving Riches is high enough to make my company worth investing in. The investors assign too large a likelihood that other potential investments would outperform my firm's stock. But then I develop an evil alternative research and development plan under which I have the exact same probability of achieving Riches as before but now if I fail to create a controllable AGI, an unfriendly Foom will destroy humanity. Now I can truthfully tell potential investors that it's highly unlikely any other company's stock will outperform mine.

Luke: In the case of the asteroid, the investor (me) has no ability to affect whether asteroid hits us or not. In contrast, assuming investors want to avoid risking the destruction of money, there is something they can do about it: they can simply not invest in the project to create controllable AGI, and they can encourage others not to invest in it either. If the dangerous AGI project fails to get investment, then those potential investors need not worry that the value of money — or indeed their very lives — will be destroyed by that particular AGI project.
Why would investors fund something that has a decent chance of destroying everything?

James: Consider a mine that probably contains the extremely valuable metal mithril. Alas, digging in the mine might awaken a Balrog who would then destroy our civilization. I come to you asking for money to fund my mining expedition. When you raise concerns about the Balrog, I explain that even if you don't fund me I'm probably going to get the money elsewhere, perhaps from one of the many people who don't believe in Balrogs. Plus, since you are just one of many individuals I'm hoping to get to back me, even if I never make up for losing your investment, I will almost certainly still get enough funds to dig for mithril. And even if you manage to stop my mining operation, someone else is going to eventually work the mine. At the very least, mithril's military value will eventually cause some nations to seek it. Therefore, if the mine does contain an easily-awakened Balrog we're all dead regardless of what you do. Consequently, since you have no choice but to bear the downside risk of the mine, I explain that you might as well invest some of your money with me so you can also share in the upside benefit.
If you're a self-interested investor then when deciding on whether to invest in an AGI-seeking company you don't look at (a) the probability that the company will destroy mankind, but rather consider (b) the probability of mankind being destroyed if you do invest in the company minus the probability if you don't invest in it. For most investors (b) will be zero or at least really, really small because if you don't invest in the company someone else probably will take your place, the company probably doesn't need your money to become operational, and if this one company doesn't pursue AGI another almost certainly will.

Luke:  Thanks for that clarification. Now I'd like to give you an opportunity to reply to one of your critics. In response to your book chapter, Robin Hanson wrote: "[Miller's analysis] is only as useful as the assumptions on which it is based. Miller's chosen assumptions seem to me quite extreme, and quite unlikely."
I asked Hanson if he could clarify (and be quoted), and he told me:
In [this] context the assumptions were: "a single public firm developing the entire system in one go. . . . it succeeds so quickly that there is no chance for others to react – the world is remade overnight." I would instead assume a whole AI industry, where each firm only makes a small part of the total product and competes with several other firms to make that small part. Each new innovation by each firm only adds a small fraction to the capability of that small part.
Hanson also wrote some comments about your book Singularity Rising here.
How would you reply to Hanson, regarding the economic incentives facing a business working on AGI?

James: Hanson places a very low probability on the likelihood of an intelligence explosion. And my chapter's analysis does become worthless if you sufficiently discount the possibility of such a Foom.
But in large part because of Eliezer Yudkowsky, I do think that there is a significant chance of their being an intelligence explosion in which an AGI within a matter of days goes from human-level intelligence to something orders of magnitude smarter. I described in Singularity Rising how this might happen:
Somehow, a human-level AI is created that wishes to improve its cognitive powers. The AI examines its own source code, looking for ways to augment itself. After finding a potential improvement, the original AI makes a copy of itself with the enhancement added. The original AI then tests the new AI to see if the new AI is smarter than the original one. If the new AI is indeed better, it replaces the current AI. This new AI then proceeds in turn to enhance its code. Since the new AI is smarter than the old one, it finds improvements that the old AI couldn't. This process continues to repeat itself.
Intelligence is a reflective superpower, able to turn on itself to decipher its own workings. I think that there probably exists some intelligence threshold for an AGI, whose intelligence is based on easily alterable and testable computer code, at which it could Foom.
Hanson, however, is correct that an intelligence explosion would represent a sharp break with all past technological developments and so an outside view of technology supports his position that my chosen assumptions are "quite extreme, and quite unlikely."

Luke: I think Yudkowsky would reply that there are many outside views, and some of them suggest a rapid intelligence explosion while others do not. For example in Intelligence Explosion Microeconomics he argues that an outside view with respect to brain size and hominid evolution suggests that an AGI would lead to intelligence explosion. My own preferred solution to "reference class tennis" is to take each outside view as a model for how the phenomena works, acknowledge our model uncertainty, and then use something like model combination to figure out what our posterior probability distribution should be.
Okay, last question. What are you working on now?

James: I'm co-authoring a book on better decision making with the  Center for Applied Rationality. Although the book is still in the early stages, it might be based on short, original fictional stories. I'm also probably going to be an editor on the next edition of the Singularity Hypotheses book.

Luke: Thanks, James!
The post James Miller on Unusual Incentives Facing AGI Companies appeared first on Machine Intelligence Research Institute.