Mikhail Batin
Alexey Turchin
Alisa Zhila
David Denkenberger

Artificial Intelligence in Life Extension: from Deep Learning to Superintelligence

Introduction The 2010s have shown a rapidly growing interest in Artificial Intelligence (AI) technologies  [63] . In recent years, AI has appeared in top scientific news sources, in stories that have demonstrated that AI is "smarter" than humans when it comes to playing a number of boardgames  [89]  and word games  [61] , thus revealing that AI is approaching a revolutionary point in its development. Investments in AI-related projects have increased dramatically in the last few years. Global AI startup financing reached US$5 billion in 2016  [76] . The current market of AI in medicine is estimated at US$1.1 billion and is expected to grow to US$9.1 billion in the next decade  [118] . Major IT companies including Google, Facebook, IBM, Intel, and Microsoft nearly simultaneously established biomedical subdivisions because their leadership sees great potential for AI in healthcare. Based on the current rate of development, it is probable that AI will become a revolutionary technology in healthcare in the upcoming decades. AI has the potential to have the greatest impact on the human life span through life-extension technologies, but the means are underexplored. In this article we investigate which AI technologies in healthcare are likely to provide the best results in the quest for increased life expectancy. There is a great number of publications about the practical applications of existing AI in medicine and healthcare. A recent review performed by Ching, et al.  [24]  describes opportunities and obstacles for the applications of deep learning in medicine. Unlike their review, ours concentrates on expected applications of different stages of AI development to fight the main cause of death in humans, aging. We demonstrate how gradual evolution of AI in medicine will result in medically oriented beneficial superintelligence able to produce indefinite life extension. The considered time span also distinguishes this work from other analyses of benevolent AI, such as  [16]  and  [58] , which immediately jump to the stage of superintelligence, when AI will, by definition, be able to solve most or all of our problems. As AI is constantly evolving, we should determine how to use it most efficiently during each stage of its development and look at the period between now and superintelligence. Only by doing this will we be able to achieve the longest possible life extension for currently-living human beings. In this article we outline a path for the application of AI to life extension that yields increasing gains at each step. We show that analysis of aging biomarkers and geroprotectors with the use of narrow AI will make the largest impact on human life expectancy with a relatively small investment. We also show how an increasing amount of an individual's healthcare data collected via wearable devices ("wearables") will feed the datacrunching ability of AI and provide constant personalized monitoring of that individual's health on ever-deeper levels, thus preventing illness at earlier stages as well as repairing age-related damage. We also demonstrate how AI-powered robotics will gradually become inner parts of the human body, resulting in cyborgization and high survivability. Our final point of interest is integration of AI with the human brain via neuroimplants to enable mind uploading. See table  1  for an outline of the expected evolution of the application of medical AI in life extension. The growth of AI's ability for independent research will be increasingly helpful in finding new technologies to lower human mortality until AI reaches the stage of self-improvement. We expect that the development of medical AI will at least partly offset the existential AI risk  [16]  via intrinsic orientation of medical AI on human benefit and AI's closer integration with humans via brain implants (see section 7.2). This article is conceptually similar to the report on the expected development of military AI  [28] , in which the same three levels of the future of AI are considered. The idea that AI will help us to make large gains in life expectancy has been explored in works of futurists Ray Kurzweil  [58]  and Robert A. Freitas Jr.  [36] , among others. This paper is structured as follows. In section 2, we review the expected progress in AI, the levels of development of AI, and the predicted timeline for the corresponding advances. In section 3, we review the current applications of AI to life extension, as developed by select startups and academic projects. Prospective near-future applications of AI to life extension and antiaging are outlined in section 4, which covers research that is yet to be transferred from academia to the lifeextension industry. The expected effect of artificial general intelligence (AGI) on life extension and applications that it will enable are discussed in section 5. The more distant future of AI, including superintelligence and its effect on life expectancy, is outlined in section 6. In section 7, we conclude our overview with a discussion of the best strategies for using AI to maximize the life span of the currently living generation.
 AI development in the twenty-first century 2.1 AI development pace Predictions about the development of AI have been complicated by AI "winters," periods of decline in funding and enthusiasm due to the lack of breakthroughs. Despite past "winters," the advancement of AI technologies has skyrocketed in recent years. We are living in a very exciting moment, considering the overall rise in enthusiasm for AI. According to one survey  [16] , a majority of scientists believe that human-level AI, then superintelligence, will be achieved before the end of the twenty-first century. The current moment (2016-2017), is a period of accelerated AI development, fueled partly by the hype surrounding neural networks and machine   If we extrapolate current trends in the performance and capacity of neural networks, infrahuman (that is able to most things that can do ordinary human being and may work as a robotic brain; but some complex creative activity is still beyond its abilities). AI could be achieved as soon as the 2020s  [93] . A recent, large poll of AI scientists  [41]  shows that AI is expected to be able to master human language around 2026 and, with 50 percent confidence, that machines will exceed humans in every task by 2062. If AGI appears soon enough, its impact will overshadow that of the slower, decade-long research in geroprotectors described below, and thus make them obsolete even before their fruition, as AGI will provide better solutions. Yet we cannot rely on the early AGI scenario, as AI prediction is known to be difficult. In any case, two possible scenarios are: -AGI will be achieved in the coming two decades; -AGI will be achieved by the end of the twenty-first century. There is a big practical difference between these two scenarios. In the first case, the majority of people living today will be able to use AI for almost indefinite life extension. In the second case, most currently living people will be able to enjoy the benefits of AGI only if a huge effort is made to take advantage of all intermediate life-extension technologies to help the current population survive to see AGI achieved. Aubrey de Grey named the situation of improving life expectancy rate equal to the passage of time "longevity escape velocity"  [4] . The result would be indefinite life expectancy (ignoring accidents, global catastrophes, etc.). In this paper we show that AI is the main "game changer" that will help currently living people reach longevity escape velocity, as its effects over time will outweigh other known means of life extension. AI is the most rapidly developing technology, and it affects and accelerates the development of all other lifeextension technologies. The exponential growth of AI, which is now doubling with a period of one year, according to [33], will potentially be able to compensate for the exponential growth of the probability of human death because of aging, which doubles every seven years  [37] , but there is large lag of implementation of medical AI technology. However, it is possible that AI growth will slow down, as it happened several times before during AI winters, and will be sigmoidal. In  [15] , Nick Bostrom shows that each day of delay in the achievement of superintelligent AI, which would reverse aging, costs 100 thousand human lives. The pace of the AI progress is very uncertain but for the purpose of this article, we are going to talk about stages of AI development in a way that is agnostic to timelines.
 The three levels of the future of AI development In this section we clarify and enhance the classification of the levels of the prospective AI. These levels are often mixed in AI discussion, which leads to confusion. Narrow AI (weak AI) is the level of a computer program that achieves above-human performance in a specific, narrow task  [16] . For example, the tasks of MRI scan recognition and facial recognition require two differently trained systems, although the underlying learning mechanism may be the same. Most existing AI systems are considered narrow AI. The number of such programs is growing rapidly due to the success of machine learning and neural networks. The difference between narrow AI and conventional computer programs is the ability of the former to learn. Autonomous cars employ a good example narrow AI. Such AI systems do not have full human capacity, particularly in generalization. Additionally, the majority of contemporary AI systems need ongoing human supervision. AGI (human-level AI) is AI at the level of human intelligence in many areas. For example, there would likely be communication in natural language, understanding the context of most situations, as well as performing most of the intellectual tasks that humans are able to perform. Philosophical questions about the possibility of consciousness in AI are outside the scope of this pragmatic definition. Ability to self-improve is an obvious consequence of this level of AI development. As a result, according to Nick Bostrom  [16] , an era of human-level AI will be brief, as AGI with self-improving abilities will soon evolve superintelligence. Robin Hanson  [45]  adheres to the view that computer modelsemulations-of the human brain will dominate in the future. Superintelligence is the level at which AI will supersede humans in all aspects, overtaking the intelligence of the entirety of human civilization. It will be able to govern the world, make scientific discoveries, launch space exploration, and create accurate simulations of the human past. Bostrom  [16] , Yampolskiy  [113] , Yudkowsky  [114] , and many other scientists expect its eventual appearance.   [51] . Many startups promise almost unbelievable feats. A collection of press releases for such companies comprises hundreds of pages of breathtaking announcements and lengthy enumerations, but most projects vanish within a few years as the survival rate of startups is low  [38] . In order to attract investors, promises are often exaggerated. However, these promises may be used to measure general trends and expectations in the industry. We can expect investment in AI to grow in the next years if a new AI winter does not occur. The healthcare sector is the largest potential source of funding for AI  [11] , as it is still a "deficit market" due to a large, unmet demand for better health.
 AI in medical research Even in scientific research, it is necessary to distinguish between "advertising" statements that often exaggerate achievements and real practical achievements. As to the former, in 2009 it was stated that a robot called Adam was able to formulate hypotheses and conduct experiments on the yeast genome  [95] . But there were no subsequent publications on this device. On the other hand, robots have indeed made substantial contributions to the automation of laboratory studies. For instance, robotic manipulators have automated repetitive operations with test tubes  [13] . Among the recent practical applications of AI is the use of artificial neural networks for visual recognition of brain scans, including reconstruction of the relationships between biological neurons in brain connections  [25] . Several companies are using AI to accelerate their research: Gero (formerly known as Quantum Pharmaceuticals) employs the methods of physical kinetics and the modern theory of dynamical systems to model aging processes in complex biological regulatory networks  [27]  aiming to develop novel anti-aging therapies. To control the health effects of the future drugs Gero team has applied a deep convolutional neural network (CNN) to time series representing human locomotor activity from wearable devices, which allowed to produce a digital biomarker of aging  [28] . This biomarker now serves as the scientific basis for Gero lifespan/health risks estimation app  1  and could be used as a metrics of health outcomes for wellness and life insurance industries. Deep Genomics is working on a system that will allow studying, predicting, and interpreting how genetic 1 https://itunes.apple.com/us/app/gero-lifespan/id1222911907 variations change important cellular processes such as transcription, splicing, and so on.  [119] . Atomwise aims to reduce the cost of new-drug development through the use of a supercomputer and a database of molecular structures to predict which versions of a potential drug will work and which will not.  [120] . There are many other companies and scientific groups that use AI to accelerate their medical research, and competition is fierce. Not all of them will survive.
 AI in diagnosis Claims that AI has outperformed humans in various narrow areas of healthcare have appeared since the 1980s  [18] . In the early days, such claims mostly referred to expert systems that were popular at the time. It was difficult to translate such success into wider practice, though-and this scaling issue has plagued AI research from the beginning. Yet humans are not much better. It was found that in 88% of cases a second opinion gives a different diagnosis  [104] . Of course, this estimate may be unrepresentative, as only uncertain cases require additional evaluation, yet it demonstrates uncertainty in human diagnostics. In April 2016, it was stressed by Mark Zuckerberg that machine learning helps to make diagnosis more accurate, inexpensive, and, perhaps most important, quick  [46] . For example, an app that tracks changes in moles based on photos taken with a cell-phone camera can replace expensive visits to a doctor. This software, Total Body Photography, analyzes photos of moles in comparison with images of 50 million malignant moles using Israeli image recognition technology  [88] . AI will be able to simulate biological processes in the human body and use the resulting models for prediction and diagnosis. This is done by using "big data"-that is, by combining a vast amount of data collected from wearables with the extensive data accumulated in previous medical practice. In 2016, IBM bought several corporations that had extensive data on an enormous number of patients. One of these, Truven, which alone has hundreds of millions of medical records, has been bought for US$2.6 billion  [26] . AI is also working with text and natural language, which helps to handle scientific papers, medical records, and patient complaints, but it still has considerable difficulty understanding human language  [7] . IBM Watson for Oncology is a cognitivecomputing system that can answer questions formulated in a natural language (that is, in a human language). It has access to various sources of data: encyclopedias, databases of scientific articles, and knowledge ontologies. Thanks to its huge computing power and preprocessed sources, it can give accurate answers questions it is asked. Since 2013, IBM Watson has been used at the Memorial Sloan Kettering Cancer Center to facilitate decision-making about treatment of patients with lung cancer. Its database is constantly updated with new disease records. IBM Medical Sieve "is an ambitious long-term exploratory grand challenge project to build a next generation cognitive assistant with advanced multimodal analytics, clinical knowledge and reasoning capabilities that is qualified to assist in clinical decision making in radiology and cardiology"  [50] . Google DeepMind (DM) Health is a Google DeepMind subproject that applies AI technology to healthcare  [29] . In collaboration with the University College London Hospital, DM will be involved in an algorithm-development project for automated distinguishing between healthy and cancerous tissues in the head and neck area. Babylon Health (iOS, Android) is a mobile application that allows a user to have an online consultation with a British or Irish doctor  [5] . Turbine.ai is a team of scientists that formulate personalized methods of treatment for any type of cancer based on AI.  [98] . Insilico Medicine is another startup working on the implementation of deep learning in drug discovery.
 AI in bioinformatics and modeling of living organisms Often artificial intelligence is thought of as something that people have not experienced yet, and when it becomes familiar and accessible, it stops being perceived as AI and is perceived more as a mere "computational method." A set of such computational methods in biology is called bioinformatics. The field of bioinformatics consists of analysis of the genome, its changes, genome linking to proteins, conformation of proteins, and the evolution of living organisms in general. The next step in the development of bioinformatics is simulation of living organisms. To make this happen, an entity needs data on cellular processes, huge computing power, and adequate biological models. One of the first computer models of a living cell was created at Stanford in 2012  [54] . It was the simplest mycoplasma, with only 525 genes. However, Craig Venter, who was working with the same mycoplasma in 2015, recognized that the functions of some 90 genes were unknown, and therefore the completeness of the model is in question  [49] . Venter managed to create a viable synthetic organism (Mycoplasma mycoides JCVI-syn3.0), whose genome consists of 473 genes, but 149 of them were not fully understood  [117] . Cell modeling cannot always be accurate, as it has many levels of uncertainty, starting from the quantum level and protein folding, Brownian motion, and so on. Quantum computers may help with protein-folding modeling in the future. So far, the most advanced simulation of a multicellular organism has been carried out on the Caenorhabditis elegans worm [77]. The simulation includes a model of its "brain," which consists of 302 neurons, and the connectome of which has been known for a long time  [110] . Some of its functions have been put into the model, but full, correct modeling of its behavior has not been achieved yet. Modeling of a human cell is much more complex than modeling of a mycoplasma cell because it includes up to 40 times more genes, but such a model will allow medication testing through computer simulation. It will also allow preclinical testing on a variety of substances as well as determining the positive effects of a particular medication positive and how it works. Any divergence from an experiment will contribute to the model's improvement. For now, "organ-on-a-chip" works as a proxy for in vitro and in silico research  [80] . The next stage of this approach will be the modeling of a particular human organs and then full body based on its genome, epigenome, and data from medical analysis. Such a model will enable precise calculation and definition of a medical intervention when required  [10] . Big companies are interested in cell modeling as well. Chan Zuckerberg Biohub, for instance, has begun work on the atlas of all human cells [121].
 Merging computational biology, cell programming, and AI Cell programming is akin to bionanorobotics: making a cell perform more and more complex tasks, including calculations, guided moving, and most importantly, protein creation in specified locations. One of the main applications of the technology is drug delivery to fight cancer. However, to program cells, one needs to process enormous amount of data about their DNA networks. This is where AI and machine learning come in. The Cellos project  [47] , which was presented to the public in 2016, performs DNA-design automation for new living organisms. It can calculate (and then synthesize) a DNA sequence that corresponds to a certain function carried out for specified cell types. Boolean logic (commands such as "AND" and "OR") can be used in this function. Molecula Maxima  [69]  is a similar platform, which is positioned as a programming language for genetic engineering. It is worth mentioning DNA origami technology  [6] , which allows the construction of different microscopic mechanisms from DNA. It is enabled through a very powerful system of computer-aided design that can decompose a designed project into its component elements (blocks), and then write the DNA code that will guide self-assembly into a predetermined shape.
 AI, wearables, and big data There are hundreds of different medically oriented wearables on the market, the explosion of which began several years ago with fitness trackers such as Fitbit. Other wearables include professional medical monitoring devices, such as devices that track heart abnormalities. The BioStampRC sensor [122] is a patch that can be glued to different parts of a body, and it collects various kinds of data and automatically loads them into the cloud. Similar to wearables are medical implants. One example is an implanted cardiac defibrillator (ICD), which was been used to give an electric shock to restart the heart and save a soccer player on the field  [21] . It might be possible to improve the situation by introducing AI trained on large amounts of data in order to define the probabilities of successful ICD therapy for a particular patient in a particular case. Final Frontier Medical Devices produces devices that can diagnose 90% of emergency situations at home.  [109] . Nimb is a wearable ring for requesting emergency help.  [123] . Wearables can collect chemical signals from the skin or electrical signals from the brain and heart. The next stage in the development of wearables will involve integrating them more closely with the human body and reducing their size. Wearables have improved clinical trials by constantly measuring numerous parameters as well as tracking whether drugs have been taken. AiCure requires taking a photo of a pill in a patient's mouth  [124] . A general trend is that smartphones "absorb" specialized gadget functions. This has happened with fitness trackers, which are currently being replaced by the Argus app. Current smartphones can measure blood oxygenation with their camera, replacing a US$50 monitoring gadget with a US$5 app. Besides the cost savings, the body space limits the number wearables that can be used at one time (setting aside the inconvenience of keeping multiple devices charged and updated). Hence, incorporating all wearables into one device is reasonable. The future universal device will likely combine a smartphone, medical device, and brain-computer interface, and might well take a wearable form such as glasses (Google Glass, for example) or a necklace. Wearables will work together with different safety systems, integrating with infrastructure and optimizing the performance of smart homes  [12] , self-driving cars, robot police, surveillance, drones, and the "Internet of things," providing a ubiquitous safety and healthcare net. Even toilets can be made "smart," analyzing biological material every time you visit them  [91] ,  [116] . Google has already patented a smart bathroom  [59] .
 The problem of research data verification: blockchain and evidence systems There is a reproducibility crisis medicine  [53] . It is explained by a number of statistical biases as well as fraud and market pressure. Life-extension studies are especially susceptible to fraud, as people are willing to pay for "youth," and it is not easy to make objective measurements in such studies. By being able to work through a large amount of patient data, AI will increase the reliability of results. Experiment automation, experiment-procedure recording, and the use of blockchain  [70]  to keep records secure could simplify verification processes and reduce bias and fraud in the field.
 Prospective applications of AI in aging research
 Fighting aging as the most efficient means for life extension It is widely understood nowadays that the purpose of general healthcare is not only to treat certain diseases but also to prolong healthy human life span. Different applications of AI in healthcare have different effects on life expectancy. For example, fighting rare diseases or advanced stages of cancer will not yield much increase in total life expectancy over the entire population. The main causes of death in the US are circulatory diseases (23.1% cardiac deaths, 5.1% stroke deaths), cancer (22.5%), chronic lower respiratory disease (5.6%), and Alzheimer's disease (3.6%). Combined, these conditions cause 59.9% of all deaths in the United States  [44] . The probability of these diseases increases exponentially according to the Gompertz law of mortality  [66, 67] . More than 75% of all deaths happen to people of 65 years of age or older  [40] . As a result, some authors  [105] ,  [115]  say that aging is the main cause of death and that if we are able to slow the aging process, we will lower the probability of agerelated diseases and increase the healthy life span. Experiments show that even simple interventions can slow the aging process and thus delay the onset of deadly diseases in and extend the healthy life span of the C. elegans worm  [20] , mice  [66] , and rats  [87] . These life-extension experiments on animals have involved relatively simple interventions, such as administering long-known drugs (metformin or rapamycin, for example) or restricting caloric intake. Such life-extending drugs are called geroprotectors  [71] . Unfortunately, studies of the life-extending effects of geroprotectors on humans are scarce, although similar interventions have often been used for other diseases (treating diabetes with metformin, for example), hence proving their safety. Although such studies could have begun long ago, this has not happened, because of a number of social and economic reasons. Naturally, such experiments would require a lot of time (longitudinal experiments take decades) and test groups would need to be large. Yet there is not the luxury of decades and centuries for classical experiments, as people are dying now, during our lifetime. There is a need to find ways to extend human life-and prove that these inventions work-in a shorter time. A well-recognized way to do this is to find aging biomarkers that will track that aging is slowing before all participants of an experiment die. In short, to slow the aging process, we must find efficient geroprotectors and combinations of geroprotectors; to prove that they work, we need to have independently verified aging biomarkers. There are many other advanced ideas in the fight against aging, including gene therapy, stem cell research, and Strategies for Engineered Negligible Senescence (SENS)  [27] . However, in this section we will limit ourselves to AI-based methods for creating efficient geroprotectors and biomarkers. There has been only one known attempt to use AI to predict aging biomarkers, which involved training neural networks on a large age-labeled sample of blood tests  [82] .
 Aging biomarkers as a computational problem Aging biomarkers are quantitative characteristics that predict the future life expectancy of an organism based on its current state  [72] . They can be normalized to a "biological age," which can be older or younger than the actual age. Future life expectancy is the difference between the average median life expectancy for a species  2  and the biological age of an individual. Different aging biomarkers have different predictive power  [64] . For example, gray hair is a marker of aging, but it has low correlation with mortality. Good aging biomarkers should be causally connected to a potential cause of death. Hair color is not causally connected to a potential cause of death, as one could dye one's hair without affecting life expectancy. In contrast, blood pressure and a number of genetic mutations are causally connected with mortality. Thus, they are better biomarkers for aging. Since aging is a complex process, it cannot be expressed by a single number; a large array of parameters is needed to represent it. Aging biomarkers should also be reversible: if the aging process has been reversed, the biomarkers' respective characteristics should change correspondingly (e.g., decrease in number). There are two ways to find biomarkers: modeling of aging processes, and statistics. As a side note, one could also measure small changes in the Gompertz curve of mortality, that is, use the number of deaths in a population as an aging biomarker  [79] . However, to observe them, information about millions of people would be required. With the help of modern wearables, it is possible to record all the drugs and treatments received by a patient. A huge number of patient records, along with corresponding data on personal genetics, physical movement, and lifetime behavioral activity, could be collected and centralized. This would result in a cohort study with better information supply and stronger probative value. Collecting and interpreting this information would likely require powerful AI. One plausible AI scenario in biomarker detection is the use of unsupervised machine learning over a large set of biomedical parameters that may lead to the discovery of groups of parameters that correlate with biological aging. Further, parameter-variance analysis will help to detect real aging biomarkers. For example, the company Gero focuses on gene-stability networks  [56] . Another application of AI in the fight against aging is in creating completely new geroprotectors by analyzing cell models, aging models, and molecular properties. Rather than drugs, the geroprotectors could be genetic interventions, that is, insertions of new genes or alterations in the expressions of existing genes (epigenomics). Five hundred thousand British senior citizens have donated their blood and anonymized their healthcare data for use by Biobank, which is now sequencing their genomes. Biobank will provide open access to all the resulting data, which will become an enormous data set for various forms of machine-learning research  [125] . Especially promising is the search for genetic networks of aging. Similar projects are taking place in Iceland  [81]  and Estonia.
 Geroprotector's combinatorial explosion A number of medications can extend the life of a mouse by slowing down its aging processes  [57] . Most of these medications, however, yield only a 10-15% increase in life span. In humans such medications would yield even less, perhaps around 5%, as longer lives are more difficult to extend, and they respond less to known geroprotectors. But what if several geroprotectors are combined? Results of a few studies on mice are promising, as they show a multiplication of effects  [96] . Recent research used a sophisticated testing algorithm to identify three drugs that yield maximum life extension in worms and flies  [31] . While that algorithm was designed manually, we expect that the best testing scheme would involve AI-aided design of a range of algorithm alternatives. Although combining certain pairs of geroprotectors works well enough, some geroprotectors are incompatible with one another. Moreover, combining them greatly reduces their effects. Hence, pairwise testing of geroprotector combinations is needed to begin with, followed by larger combinations. To test all combinations of 10 geroprotectors would require 1024 experiments, and for 20 geroprotectors the number of experiments would be over a million, and that is for a single dosage rate for each geroprotector. This is virtually impossible, as there financing has been unsuccessful for even simple testing of one combination on mice (see lifespan.io campaign  [126] ). The problem of searching in an enormous space is similar to that of playing a complex board game with a huge search space, such as Go. The recent success of AlphaGo  [127]  promises that such a search could be simplified. Consequently, a much smaller number of experiments would need to be run to determine an optimal geroprotector combination. The underlying principle of AlphaGo is that the most promising combinations are selected by a neural network trained on a large number of previous games. Similarly, a neural network can be trained to predict the biological effects of chemicals based on knowledge of their properties obtained from a comprehensive library of substances. A similar computational approach is used for drug discovery  [92]  and toxicity forecasting  [103] . Toxcast is a large US-government-sponsored program designed to use machine learning to predict the toxicity of different chemicals  [86] . To increase the number of useful outcomes of an experiment, it is also necessary to record a vast number of various vital parameter measurements of an organism (for instance, blood composition, physical movement, EEG readings) during the process of geroprotector testing. This would allow the discovery of aging biomarkers during geroprotector testing. Generally, the geroprotector-identification problem can be reduced to the task of finding a global minimum of a function of ten (or more) variables. A number of efficient machine-learning algorithms are suited for such a task. The search for aging biomarkers can be pursued in a similar manner. From the mathematical point of view, it is a search for the global minimum of the function of many properties of an organism. The same process can also be used to calculate specific gene interventions for an individual human, in view of the genome characteristics, age, and biomarkers. Activities in this area are carried out by Gero, Calico, the Buch Institute  [19] , and others. João Pedro de Magalhães has used random-forest machine learning to predict the properties of life-extending compounds  [9] . Additionally, several projects are searching in large combination spaces by using neural networks designed for other tasks: -Project AtomNet  [3]  predicts the properties of chemical materials using convolutional neural networks; -E. Pyzer-Knapp et al.  [83]  are using a multilayer neural network to predict the electrical properties of new molecules; -L. Rampasek and A. Goldenberg  [84]  are reviewing applications of neural-network project TensorFlow by Google in computational biology; -K. Myint and X.-Q. Xie are predicting ligand properties using a fingerprint-based neural network  [74] .
 AI, aging, and personalized medicine Aging can be viewed as the accumulation of errors and lack of adequate regulation in a body by repair mechanisms and the immune system  [37] . Hence, in the fight against aging, additional regulation is needed in the form of medical therapy. Medical therapy consists of tests (for instance, blood work, blood pressure readings, medical scans), hypothesizing about causes of disease (diagnosis), medical intervention, and in the case of an incorrect hypothesis, subsequent correction based on new observations. This process is similar to the scientific method, and at its core it is an information-based process, that is, a process of solving a particular computational task. This means that it will benefit from more data and more intelligent processing, followed by a precise and targeted intervention. Therefore, to cure a disease or rejuvenate a body, it is helpful to collect a large amount of information from that body, in order to construct a detailed model of it. This will enable calculations for the genetic interventions that will lead to recovery and functional improvement. It is now possible to obtain large amounts of data on a body via full genome sequencing, thousands of parameters of blood analysis, and analysis of the transcriptome, metabolome, and other similar "omics" (that is complex quantitative description of functions and statistics of a type of organism's elements). This is achieved through continuous monitoring of food intake, physical activity, and heart parameters via ECG, various scans, and digital tomography. The rapid decline in the cost of all these procedures (US$999 in 2016 for complete sequencing of a genome  [78] ) has led to individual humans becoming sources of big data. Now we are faced with the question of how to interpret these data to produce the best effects on human health by not only diagnosing existing illnesses but also by predicting future illnesses and creating personalized aging profiles. For this reason, there needs to be better means to derive meaningful conclusions from this vast amount of data. In the past, the following situation was typical: a patient complains to a doctor about various aches and, after having their blood pressure and temperature measured, receives treatment with a single prescribed medication. In this case the information exchange between the patient and the doctor consisted of "just a few bytes" and some intuitive impressions of the doctor. However, nowadays the information exchange may consist of gigabytes of information at the same cost. For the processing of this data stream, powerful data crunch techniques are required. During aging, a body gradually accumulates errors, and its natural repair systems begin to fail. The information theory of aging could be designed to enable therapies to correct all these errors, and this idea is at the core of the Strategies for Engineered Negligible Senescence (SENS) project  [27] . AI may help humans to model aging by creating a complex causal map of aging processes in a body  [90]  and then personalizing the model. Naturally, an organism's body is able to solve most of its problems locally without sending information outside: cells know what to repair, and higher-level attention is needed only when they fail locally. An aging body fails to solve its problems locally. Therefore, it may be reasonable neither to extract information from the body nor to direct therapy into the body, but rather to introduce "AI helpers" inside the body, where they can help solve problems as they appear. Implants and future nanomedicine will be employed along these lines. Another solution to the "messy problem of aging" is growing completely new body parts and full bodies. However, designing the immunogenic properties of such parts and solving a complex "connection problem" will require analysis of large amounts of information, which will only be feasible if AI is employed.
 Narrow AI in medical-cost reduction and affordable healthcare Efficient and affordable healthcare will be essential to a global increase in life expectancy. Cheap mobile phones solved the communication problem at the global scale by operating as a standard solution. A similar kind of solution must be sought in healthcare. High-quality healthcare is very expensive. Nursing, hospitals, drugs, tests, insurance, and highly paid specialists all cost much money, and as a result, advanced healthcare is out of reach for many people. AI will provide less expensive services and make them available to larger population groups in developing countries. Just as generic drugs can be taken in place of expensive brand-name drugs, an AI-powered consultation could provide diagnostics for people who cannot afford a doctor. Many people-for instance, those who search the Internet for answers to their medical questions-may be less reluctant to consult an AI-powered specialist than a real doctor. The following instruments will make AI-based healthcare an inexpensive alternative to hospitals: -AI chatbots, such as the Babylon app [5]; -Smartphones as a universal diagnostic implement (they can be used to monitor heart rate, diet, physical activity, oxygen saturation, mole changes, and so on); -Home delivery of cheap generic drugs; -Web-based medical expert systems.
 Effects of narrow AI on life extension Narrow AI will help unleash the full potential of life extension, leading to dramatically slower aging. If humans did not age, they could live hundreds of years despite accidents (If we exclude age-dependent component of mortality by extrapolating of minimal probability of death found in 10 years old American girls, which is 0.000084 for a year  [1] , we will get life expectancy of 5925 years. But increasing probability of death with age lowers it to 81. Most of this death probability increase comes from biological aging.) Yet introduction of narrow AI into effective medical practice could take much longer than related advances in research labs, possibly decades. The present era of narrow AI might be long, lasting until 2075 by pessimistic predictions  [73] . However, this time can be spent usefully, exploring aging biomarkers and geroprotector combinations. For those who are not able to survive until the arrival of radical life-extension technologies, narrow AI may still play an important role by providing two main backup options: cryonics and digital immortality. In cryonics, AI applications may, via wearables, warn a patient's cryonics organization of the impending death of that patient. Cryopreservation could be called plan B, while plan A is to survive until the implantation of life-extension technology. Digital immortality  [107]  is the concept of preserving a human being's data so that future AI will be able to reconstruct his or her model using DNA, video recordings, and additional data gleaned from such sources as social networks. It depends on certain assumptions about AI's capabilities, amounts of required information, and the nature of human identity. AI could help to collect and preserve data for digital immortality and perform initial analysis of that data. Digital immortality is plan C in achieving radical life extension. An early arrival of advanced forms of AI may make these three approaches obsolete before they are implemented.
 Prospective applications of AGI to life extension
 Personal robot physician AGI may appear in the form of a human-mind upload  [23] ,  [45] , or as an infrahuman robotic brain  [17]  capable of performing most human tasks. It will be Turing complete  [112] , meaning that it will be able to interact conversationally approximately as well as a human. There are numerous ways in which AGI may be applied to life extension. In this section, we will explore those that are likely to provide the biggest gains in life expectancy. Cheap and efficient AGI will enable accessible and predictive personal healthcare. A plausible example is an AI-based personal assistant that will be a combination of a healthcare researcher and personal physician and will be able to provide personal treatment and early response to symptoms. It will constantly monitor an individual's aging biomarkers and other life parameters, allowing daily therapy adjustments. A patient will no longer need to visit a clinic, get a prescription, have it filled at a pharmacy, remember to take drugs at prescribed times, try to determine whether her or she is feeling better, and so on. A personal robot will simply utilize data gathered from wearable monitoring systems to determine an ideal drug combination, order it to be delivered, and then prompt the patient to take a pill. The process of diagnosis and cure will be as effortless and automated as an upgrade of antivirus software on a personal computer. The ability of AGI to comprehend human language will lead to the possibility of "artificial scientists" that are able to formulate hypotheses, organize experiments, and publish results as scientific papers with less and less help from humans. Combined with robotized labs and less expensive equipment manufacturing, AGI will accelerate scientific research in all fields, including life extension. Domestic medical robots and wearables will automate clinical trials, reducing costs and accelerating drug discovery by collecting data for clinical trails. Currently, a clinical trial may cost hundreds of millions of dollars because of legal and organizational issues. Home robots will record patient activity, automating clinical trials and making them independent of large medical companies via decentralization, which will reduce their costs and improve data objectivity. Robotic drones with drugs and defibrillators will provide assistance to people whose wearable systems report an emergency. Domestic robots will monitor the health of a family, help with treatment, monitor medicine consumption, act as physical-exercise instructors, and predict disease. Additionally, they will provide companionship for the elderly, which will also increase life span.
 Integration of monitoring systems into human bodies and nanomedicine A person's immune system maintains information on such parameters as locations of body inflammation and the types of viruses it is equipped to neutralize. This information is beyond the control of human consciousness. The immune system can be trained with vaccines, but information exchange between humans and immune systems is limited. If a person could read the immune system's information and upload new information into the system, then it would be possible to cure a large range of ailments, including autoimmune diseases, infections, organ failure, tumors, and tissue senescence. Ray Kurzweil expects communication to appear in the 2020s  [85] . The process will be similar to current computerized automobile diagnostics. A system of communication between an organism's immune system and a computer can be called a "humoral interface" and would have much in common with a neurointerface. It could be created with some form of nano-or biotechnology, such as computer-programmed cells. The next step in this direction is artificial human immune system management. Such a system may consist of biological organisms, an individual's own upgraded cells  [30] , or micro robots circulating in an organism's blood. The following are the expected levels of a nanotechnology-based upgrade of the human body: 1) In the first stage, the system will monitor emerging diseases; 2) In the second stage, the system will assist in treatment by killing bacteria, viruses, and cancer cells, and by repairing vascular injuries; 3) In the advanced stages, the system will constantly carry out body repair and treatment of aging; 4) In the final stage, these systems will transform into nanomachines that will replace human cells, making the human body completely artificial and immortal. This will likely only happen when AI reaches the superhuman level.
 "The Upgrade Net": a path to superintelligence through a network of self-improving humans and humanlike AI Systems As Elon Musk famously tweeted, "Humans must merge with machines or become irrelevant in AI age"  [55] . Such a merger would require a powerful braincomputer interface (BCI), and we think that the best way to achieve this is through the implementation of a personal AI health assistant, which would be integrated into human bodies and brains and focused on preserving human lives. Musk has also stated  [102]  that he wants to commercialize the AI health assistant with his Neuralink project. Neuralink will begin by using a simple BCI to treat depression and other mental illnesses. A simple BCI may be used to control human emotions, preventing mental-state-dependent types of violence such as road rage and suicide. This will provide experience that can be directed toward curing mental diseases with BCI, and eventually proceeding to a stage of augmented humans, who could later be connected into a network of selfimproving humans. In our opinion, there is another way of building a network of self-improving humans, and it starts with the creation of a medical social network: First, new type of patient organizations  [42]  will need to be established to connect people who are interested in the fight against aging  [128] . These organizations will essentially operate as social networks for information exchange, mutual support, clinical trials, crowdfunding, data collection for digital immortality, civil science, aid in cryopreservation, and political action. Individual biohackers also could play important role by self experimentation, like Elizabeth Parrish: they could take higher risk experiments on themselves without legal restriction and costs  [68] . The next step will be the creation of a network for direct interaction between the brains of human participants, a so-called neuroweb  [60] . Informationtransmission mechanisms may be implemented using weak AI systems. The result of such a network will effectively be a collective brain. Direct brain connection may be confusing and inefficient, so a kind of AI firewall may be required to control access to the information that an individual wants to share. Also, an AI dispatcher may be needed to facilitate conversation by remembering conversation's lines, providing relevant links, illustrating ideas, and so on. At a further stage of development, an AGI-based virtual assistant connected through BCI to a human's brain may work as a form of exocortex  [14] . The ultimate step is to merge with AI, which implies blurring the boundaries between the biological brain and the computer. This is equivalent to achieving practical immortality (if no global risks will happen), because brain data will be easily backed up and, if needed, restored. Effectively, human minds and computer superintelligence will merge into a single system. At the same time, people will be able to maintain a preferred level of autonomy with regard to memory, consciousness, and learned skills  [34] ,  [101] ,  [75] . 6 Superintelligence and the distant future
 Superintelligence finally solving problems of aging and death We can use trends and polls to predict narrow AI and AGI. Superintelligence is by definition unpredictable. For expectations of its arrival and what it will be able to accomplish, we can refer to various futurists: Bostrom  [16] , Yamploskiy  [113] , Yudkowsky  [114] , Kurzweil  [58] , Vinge  [106] , and Goertzel  [39]  all depict a future dominated by global superintelligence. According to these futurists, the arrival of superhuman AI will enable solutions to the problems of aging, curing presently incurable diseases, designing universal medical nanorobots, and uploading an individual's consciousness into a computer network. In the past, it took decades to accomplish complex, globally valuable tasks such as the development of modern aeronautics, wireless communication, and noninvasive surgery; superintelligent AI will be able to solve such problems very quickly, perhaps in moments. With the arrival of superintelligent AI, achieving practical immortality for the majority of people will become feasible.
 Simultaneous creation of superintelligence and advanced nanotechnologies K. Eric Drexler's book Engines of Creation  [32]  and Robert A. Freitas Jr.'s Nanomedicine, Volume IIA: Biocompatibility  [36]  discuss nanotechnology as nanorobotics based on molecular manufacturing for medical treatment and intervention. According to Drexler, medical nanobots will:  be self-replicating;  be externally controlled;  carry onboard computers;  be capable of swarm behavior  be cell sized;  be capable of 3-D printing organic structures; and be capable of sensing their environment and navigating in it. If such nanobots arrive before AGI, they will quickly help us map the structure of the human brain and develop technology to create a very powerful supercomputer, leading to the advent of AGI. On the other hand, if AGI arrives first, it will create nanobots. The wait between nanorobotics and AGI will likely be no more than a few years. Designing the first nanobot and controlling nanorobotic swarms will be a huge computational task, itself requiring the use of available AI. When this technology matures, it may enable relatively quick (hours to weeks) and seamless replacement of living cells in a human body-with the possible exception of the neurons responsible for personal experiences-with fully controlled nanomachines by injecting a single self-replicating nanobot. Such a nanotechnological body will not age as it will be able constantly self-repair according to original plan.
 Superintelligence and the solution to the consciousness problem: identity copying On the one hand, it will be difficult to develop fullfledged AGI without first solving the problem of consciousness. On the other hand, nanotechnology and AGI will give us the means to carry out various experiments on the conscious brain and map its structure. For example, investigation of qualia is feasible through a gradual uploading process similar to the thought experiment performed by  David Chalmers [22] . This will enable detection of the brain parts and internal processes responsible for subjective experience. There are two possible scenarios: either there is no mystery here and the problem of uploading consciousness to a computer is purely informational, or consciousness has a certain substrate. This substrate could be a quantum process, continuity of causal relationships, special particles, or similar-that provides identity, and its preservation and transfer is a separate technical task. In either case, the transfer of consciousness to a new carrier is possible: an ordinary computer can be used in the first scenario; the second scenario will require a specialized computer, such as an artificial neuron or a quantum computer  [2] . This hypothetical consciousness-receptacle computer will need to be extremely resistant to damage and have advanced backing-up abilities in order to lower the risk of death.
 Using advanced forms of superintelligence for the reconstruction of the dead people Cryonics is the idea, introduced by Robert Chester Ettinger and Jean Rostand  [35]  of using low temperatures to preserve human bodies after death until it becomes possible to return them to life. Currently around 250 people are cryopreserved by three cryocompanies  [67] . At first, it was thought that bodies could be gradually unfrozen upon the appearance of appropriate technologies. Later it was thought that nanotechnology could be used to repair damage in thawing bodies  [32] . A more recent view is that bodies can be scanned without thawing  [65] . Advanced tomography  [48]  or slicing  [43]  would be employed, and the data from the scans would be entered into a computer, where the human mind would be reconstructed. Currently around 250 people are cryopreserved by three cryocompanies [122] and advanced nanotech created by AI could be used to scan and upload their minds. In addition, highly evolved superintelligence will be able to reconstruct humans who lived in the past by modeling their lives in a simulation. A reconstruction would be based on a subject's informational traces. It is called "digital immortality"  [108] . For global resurrection of the dead [123], superintelligence may perform a large-scale simulation of the past  [124] . Then, based on all the data about the past, it will reconstruct everyone who ever lived. 7 Discussion: strategies for applying AI to life extension
 Problems of AI application in healthcare In 1979, a rule-based expert system could make a diagnosis better than human doctors  [18] . Since then, decades have passed, and yet a large-scale AI revolution still has not happened in healthcare. Most modern medical systems are still based on extremely simple algorithms, for example, if the heart rate is more than X, execute Y  [8] . Brandon Ballinger  [8]  wrote that one major obstacle is the majority of "cheap" easily available datasets is not labeled, but machine-learning algorithms mostly require labeled data for training. For example, there is a lot of cardiac data, but it is not clear what disease it is associated with or what the patient's vital parameters were. To obtain labeled data, it might be necessary to conduct costly and potentially harmful experiments on humans. Currently, this problem is being approached by unsupervised learning algorithms, which do not require labeled data, but their performance is still behind that of the supervised systems. In addition, there are regulatory issues regarding the utilization of AI in healthcare, as well as disputes about risk allocation and insurance payments between startups and hospitals. AI can easily be migrated into an individual's smartphone, but getting it into a doctor's office is more complicated, not to mention the intricacies of accounting for AI in insurance payment systems. One can imagine that the modest pace of advancement of AI applications in healthcare in recent decades might be disappointing to the authors of the first edition of Artificial Intelligence in Medicine, which was published back in 1982  [97] . Yet, due to substantial increase in computing power, availability of "cheap" digitized data, advanced data-analysis algorithms, and new regulations, we finally seem to find ourselves at the dawn of the rapid development of AI in healthcare. Privacy issues regarding personal data create a tradeoff for AI development. On one hand, the greater the amount of open data, the easier it is to train AI algorithms. (Sharing one's personal health data may cause unpredictable harm to the individual, however.) On the other hand, if only anonymized data is available, important vital parameters and data points will be lost. The patient organizations discussed in section 5.3 may understand the importance of providing open access to personal data, as doing so would help train AI for healthcare.
 AI in medicine, and AI safety Issues of AI safety, on both local and global levels, are beyond the scope of this work. We want to emphasize just two points of intersection of AI in healthcare and AI safety: Medical AI is aimed at the preservation of human lives, whereas, for example, military AI is generally focused on human destruction. If we assume that AI preserves the values of its creators, medical AI should be more harmless. The development of such types of medical AI as neuroimplants will accelerate the development of AI in the form of a distributed social network consisting of self-upgrading people. Here, again, the values of such an intelligent neuroweb will be defined by the values of its participant "nodes," which should be relatively safer than other routes to AI. Also, AI based on human uploads may be less probable to go into quick unlimited selfimprovement, because of complex and opaque structure. If the orthogonality of values and intelligence thesis  [16]  has some exceptions, medical AI may be safer than military AI. On the other way, medical AI may increase the risks as it will open the way to the neuromorphic AI, which is regarded dangerous  [16] , or it will be under less control than military AI, and could run into explosive run-away self-improvement. The Upgrade Net discussed above may become a useful instrument in solving the AI safety problem, as the growing collective human intelligence could operate as a global police force, identifying potential terrorist behavior and other threats. The safety will come from intrinsic value alignment of human uploads  [94] , combined with superintelligence power of the whole net which will be able to find and prevent appearance of other types of potentially dangerous AI systems, as well as exterminate the need of creation of such systems. Turchin addressed this question in greater details in  [99] .
 Surviving to see AGI: personalized, age-dependent strategies The older a person gets, the lower his or her chances of surviving into the era of AGI and powerful life-extension technologies. Fortunately, it is not necessary to wait until superintelligence arises. In order for an individual's life expectancy to be increased indefinitely, that individual must stay alive only until the moment when average life expectancy begins increasing by more than a year each year, at which point longevity escape velocity will be achieved  [27] . However, the chances that a person will be able to benefit from life extension significantly increase if that person has better access to upcoming technologies by, for instance, living in a developed country, having financial security, or being foresighted enough to research and utilize those technologies when first available. In order to increase and spread the benefits of medical AI in the future, it will be necessary to increase people's awareness and encourage them to exercise all available means for life extension. As part of this strategy, we promote participation in patient organizations committed to fighting aging, signing up for cryonics, and sharing and collecting digital immortality data.
 Conclusion This work is an overview of the existing and prospective AI applications that the authors consider the most promising and beneficial for life extension and antiaging. We have considered a wide range of problems with the current state of the research and the industry, the most promising prospective applications of AI, and strategies to increase public awareness in order to ensure maximal life-extension opportunities for everyone. Based on related work, we have reviewed the expected stages of the development of AI in the near future, and estimated when the most advanced levels will arrive. Further, we have presented an overview of the current AI-based healthcare projects of certain for-profit companies of various scales. These projects include IBM Watson Healthcare, Google Calico, and DeepMind Health, as well as the research projects of certain academic groups and nonprofit organizations. We have shown that the exponential growth of AI's capabilities makes it more likely that AI could help fight the exponential increase of the probability of a human being's mortality over time, and that AI could help a person to reach longevity escape velocity before superintelligence is achieved. It may help millions or maybe even billions of people to "survive until immortality," and thus rescue their life from impending death. Some of the authors explored this topic in greater detail in the article "Fighting aging as an effective altruism case: the model of impact"  [100] . We have emphasized the importance of establishing patient organizations to spread awareness of the subjects of life extension, voluntary patient data collection, early adoption of medical AI technologies, and the eventual formation of a "neuroweb" with the arrival of advanced forms of AI.
 Table
 : Expected evolution of medical AI in life extension. learning. Dozens of startups are working to develop AGI, and they are attracting substantial funding. Achievements in the development of AI are doubling every year in such areas as complexity in text understanding, speech and visual recognition, and natural language conversation [33].
 3 The current applications of AI in healthcare and medical research 3.1 Growth of investments in healthcare AI general; Intel's large biotech section [52]; Microsoft's innovative cloud computations for new drug discovery; and Apple's platform for wearables and software for health monitoring. Not only big business invests in healthcare research and development; many startups are also making great strides. It is estimated that in 2016, there were 106 startups that used AI in various areas of healthcare. The number of mergers and acquisitions in healthcare AI grew from less than 20 in 2012 to nearly 70 in 2016 In 2014-16 the giants of the IT industry announced the launch of biotechnology and life-extension projects based on machine-learning techniques. Among those projects are Google's Calico, focusing on anti-aging; Facebook's Chan Zuckerberg Biohub, searching for drugs for all diseases and creating an atlas of cells for this task; IBM's Watson Health, targeting healthcare in
			 Technically the life expectancy should be at the biological age, rather than at birth as is usually quoted.

Abstract: In this paper we focus on the most efficacious AI applications for life extension and anti-aging at three expected stages of AI development: narrow AI, AGI and superintelligence. First, we overview the existing research and commercial work performed by a select number of startups and academic projects. We find that at the current stage of "narrow" AI, the most promising areas for life extension are geroprotector-combination discovery, detection of aging biomarkers, and personalized anti-aging therapy. These advances could help currently living people reach longevity escape velocity and survive until more advanced AI appears. When AI comes close to human level, the main contribution to life extension will come from AI integration with humans through brain-computer interfaces, integrated AI assistants capable of autonomously diagnosing and treating health issues, and cyber systems embedded into human bodies. Lastly, we speculate about the more remote future, when AI reaches the level of superintelligence and such life-extension methods as uploading human minds and creating nanotechnological bodies may become possible, thus lowering the probability of human death close to zero. We suggest that medical AI based superintelligence could be safer than, say, military AI, as it may help humans to evolve into part of the future superintelligence via brain augmentation, uploading, and a network of self-improving humans. Medical AI's value system is focused on human benefit. Povzetek: Prispevek opisuje najbolj učinkovite aplikacije umetne inteligence za podaljšanje življenjske in delovne dobe od klasičnega strojnega učenja do superinteligence.


The Social Science of Computerized Brains -Review of The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson (Oxford University Press, 2016)

Introduction The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson discusses the possibility of "uploading" human minds into computers. An uploaded mind or "em" (short for "emulation) is able to think more or less like a regular biological human, but with properties of computer programs and data. Ems can be copied ad infinitum and sped up or slowed down by being placed on faster or slower hardware. These capabilities create radical possibilities, which is the book's focus. The book is by far the most detailed description of the em world available. Hanson, an economist with a background in physics and artificial intelligence (AI), brings a distinctive and valuable perspective to this important topic.  The Age of Em describes one em world scenario in detail, giving only brief attention towards the end of the book (Ch. 28) to variations of this scenario. The chosen scenario is rated as the most likely one, given that uploading occurs (p.34-37). The scenario is intended as a baseline that is representative of many possible scenarios; interested analysts can introduce variations to produce still other scenarios. The book states that "Such straightforward scenario construction seems taboo among many professional futurists today", who instead favor creating "a modest number of scenarios to cover a wide range of possibilities across key axes of uncertainty and disagreement, with each scenario being story-like, archetypical in representing clusters of relevant driving forces, and describing equilibrium rather than transitory situations" (p.35), citing  Schoemaker (1995) . It is certainly true that some futures studies feature sets of story-like equilibrium scenarios, and it may well be true that some futurists disprefer presenting a single scenario, covering transition periods. However, in my experience, futurists are open to a wide range of approaches, and I see no reason why they should reject that of The Age of Em. The bulk of the book consists of descriptions of the world in which the ems The scope is encyclopedic, with attention to everything from the social relations among ems to the resistance of em buildings to earthquakes. Indeed, the book may be best treated as an encyclopedia of a particular em scenario. Reading it straight through, it can get bogged down in details. Topics are also often presented in an unintuitive order. For example, the initial creation of the em world appears in subsections of chapters 12 and 28, instead of in its logical place at the beginning of the book. A more story-like structure would make it more readable and may reduce certain cognitive biases  (Steinhardt & Shapiro, 2015) . However, if you are wondering about what some aspect of an em world might look like, odds are good that a description can be found somewhere in the book. A central premise of the book's scenario is that the em world is organized around market activity. Ems exist because they provide valuable labor. Ems with different capabilities are assigned to different tasks; groups of ems are organized into functional teams; cities are built to collocate ems for enhanced productivity. Ems are paid for their services and can spend their earnings on leisure, though competition between ems drives wages down. While these basics are broadly similar to much of today's society, the nature of ems creates some radical differences. For starters, while some ems are housed in robotic bodies to perform physical-world tasks, most live and work disembodied in virtual reality worlds. This enables them to travel at great speeds via telecommunications lines. Additionally, ems can be sped up or slowed down as needed, causing them to experience the world (and act on it) thousands or millions of times faster or slower than biological humans. Finally, ems can be copied repeatedly, which enables em teams to perform extreme multitasking. The book even proposes the concept of "spurs", which are em copies that perform a brief task and are then terminated or retired. These novel capacities enable ems to vastly outperform biological human workers. Em labor thus displaces humans while creating rapid economic growth. The book proposes that those humans who own parts of the em economy "could retain substantial wealth", but everyone else is "likely to starve" (p.336). To my eyes, this seems overly optimistic: if ems are so much smarter than humans, and if they have such a profit motive, then surely they could figure out how to trick, threaten, or otherwise entice humans into giving up their wealth. Humans would likely die out or, at best, scrape by in whatever meager existence the ems leave them with. Given these dire prospects for humans, one might question whether it would be good to create ems in the first place. Unfortunately, the book does not consider this topic in any detail. The book is pro-em, even proposing to "subsidize the development of related technologies to speed the arrival of this transition [to the em era], and to subsidize its smoothness, equality, or transparency to reduce disruptions and inequalities in that transition" (p.375). But this position is tenuous at best and quite possibly dangerous. The book's methodology is notable for its emphasis on social science. Many people who write about topics like uploading are trained in physical science, computer science, or philosophy; they thus "don't appreciate that social scientists do in fact know many useful things" (p.34). Hanson's defense of social science in this context is welcome. The book's methodology is rooted in extrapolation of current social science to em world conditions. At times, the extrapolations seem strained. For example, a claim that economic benefits of urban agglomeration would continue in an em world (p.215) cites  Morgan (2014) , which is a popular media description of Amazon's large data centers. It is true that Amazon data centers are large, and it may well be true that em cities are large for similar reasons, but the latter does not necessarily follow from the former. One can sympathize with Hanson, who is trying to present the social science of a topic that virtually no social scientists have previously touched. Indeed, the breadth of research covered is impressive; the book quietly doubles as a good literature review. However, some of the language relating the literature to the em world could have been better finessed. In other stretches, Hanson's personal tastes are apparent. This is seen, for example, in discussions of combinatorial auctions and prediction markets (p.184-188), two schemes for adapting market mechanisms for social decision making. Prediction markets in particular are a longstanding interest of Hanson's. The book's discussion of these topics says little about the em world and seems mainly oriented towards promoting them for society today. The reader gets the impression that Hanson wishes society today was more economically efficient and rational in a certain sense, and that he has embedded his hopes for a better world into his vision of ems. In conclusion, The Age of Em is a thoughtful and original work of futures studies, bringing an insightful social science perspective to the topic of mind uploading. The book covers wide ground about the nature of the em world, offering a valuable resource for anyone interested in the topic, or about futures studies more generally. The book is accessibly written and could be read by undergraduates or even advanced high school students, though experts from any discipline will also find much of interest. The book is especially worthwhile for anyone who could benefit from an overview of contemporary social science research, which the book provides in abundance. The book is not flawless. One can disagree with some of the book's arguments, including some of the important ones. The author's personal perspective is apparent; different authors would often reach different conclusions about the same topic. The organization is often scattered, and some literature is misused. However, none of these flaws take away from the significant accomplishment of describing the em world in great detail. The book offers a starting point-indeed, a baselinefor the serious study of future uploading scenarios. One can only hope that additional scholars, especially in the social sciences, follow the book's lead in taking on this important future scenario. Futures j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / f u t u r e s

Abstract: A new book by Robin Hanson, The Age of Em: Work, Love, and Life When Robots Rule the Earth, is reviewed. The Age of Em describes a future scenario in which human minds are uploaded into computers, becoming emulations or "ems". In the scenario, ems take over the global economy by running on fast computers and copying themselves to multitask. The book's core methodology is the application of current social science to this future scenario, a welcome change from the usual perspectives from physical science, computer science, and philosophy. However, in giving a wide tour of the em world, the book sometimes gets bogged down in details. Furthermore, while the book claims that the em takeover scenario would be a good thing for the world and thus should be pursued, its argument is unpersuasive. That said, the book offers by far the most detailed description of the em world available, and its scenario offers a rich baseline for future study of this important topic.


Introduction Inverse reinforcement learning (IRL) studies reinforcement learning problems where the goal is to maximize rewards in absence of precise knowledge of the reward function. The usual assumption is that there is an 'expert' who demonstrates the correct behavior with trajectories sampled from an optimal policy. A solution to the IRL problem is typically achieved by recovering the reward function  (Ng and Russell, 2000; Abbeel and Ng, 2004; Choi and Kim, 2011) . Traditionally IRL has disconnected the reward maximization from learning the reward function. In this paper, we consider IRL in an interactive setting where the agent receives information about the reward function through interaction with the environment while attempting to maximize reward. Since the information the agent receives about the reward function depends on the actions, the agent has some control over which reward function it learns, and thus is incentivized to manipulate the learning process. Our goal is to remove the incentives to manipulate, while maintaining the incentives to learn. The closest related work in the literature is cooperative IRL  (Hadfield-Menell et al., 2016)  where an agent plays a cooperative game with a human. The human starts with full knowledge of the reward function which the agent does not observe, and they are both aiming to maximize this reward function. Both can agree on prior knowledge, thus cooperative IRL overemphasizes the process of communicating the reward function from the human to the RL agent efficiently. Moreover, humans usually do not have a clear understanding of their reward function. Generally, it is unclear what kind of assumptions we can make about the human, so it is difficult to provide formal guarantees about a cooperative IRL algorithm. In this work we make the human feedback implicit: we assume that the agent somehow receives information about the reward function through its actions. Learning a reward function while maximizing reward leads to the problems caused by changes to an agent's reward function  (Armstrong, 2010; Soares et al., 2015; Orseau and Armstrong, 2016) . If the agent anticipates changes in the reward function, it will expect to achieve less reward according to the current estimated reward function. Therefore the agent has incentives to prevent this change. In our setting this can lead to the undesired effect of the agent trying to avoid information about the reward function. In this paper, we define the problem of interactive inverse reinforcement learning and analyze some of the agent's incentives. We isolate an incentive to learn from an incentive to bias and illustrate their effect on the learned reward function. Furthermore, we discuss strategies to avoid bias in the learning process. 2 Interactive Inverse Reinforcement Learning
 Interacting with POMDPs A partially observable Markov decision process without reward function (POMDP\R) µ = (S, A, O, T, O, s 0 )  (Choi and Kim, 2011 ) consists of a finite set of states S, a finite set of actions A, a finite set of observations O, a transition probability distribution T : S × A → ∆S, an observation probability distribution O : S → ∆O, and an initial state s 0 ∈ S. We model the environment as a POMDP\R. The agent interacts with the environment in cycles: in time step t, the environment is in state s t−1 ∈ S and the agent chooses an action a t ∈ A. Subsequently the environment transitions to a new state s t ∈ S drawn from the distribution T (s t | s t−1 , a t ) and the agent then receives an observation o t ∈ O drawn from the distribution O(o t | s t ). The underlying states s t−1 and s t are not directly observed by the agent. A history h t = a 1 o 1 a 2 o 2 . . . a t o t is a sequence of actions and observations. We denote the set of all histories of length t with H t := (A × O) t . A policy is a function π : (A × O) * → ∆A mapping histories to probability distributions over actions. Given a policy π and environment µ, we get a probability distribution over histories: µ π (a 1 o 1 . . . a t o t ) := s1:t∈S t t k=1 O(o k | s k )T (s k | s k−1 , a k )π(a k | a 1 o 1 . . . a k−1 o k ). The expectation with respect to the distribution µ π is denoted E π µ .
 Learning a Reward Function The agent's goal is to maximize total reward m t=1 R(o t ) up to the horizon m where R : O → R is a real-valued reward function. We assume that the environment is known to the agent. The reward function R is unknown, but there is a finite set of candidate reward functions R. The agent has to learn a reward function in the process of interacting with the environment. Importantly, the agent is evaluated according to the reward function it has learned at the end of its lifetime in time step m. Traditionally, there is assumed to be an underlying reward function R * , which the agent learns via some algorithm. Thus at the end of m turns, an agent will have a distribution P : H m → ∆R, which specifies its posterior belief P (R | h m ). This P corresponds to the learning algorithm, and is in a sense more important than R * . Indeed, R * is often a fiction, a reward function that is assumed to exist, independently of P , for ease of analysis 1 . Thus we will investigate the properties of P itself, seeing it as the more fundamental object. Example 1 (Cleaning and Cooking Robot). A household robot is trying to infer whether it should be cleaning or cooking, R := {R clean , R cook }. The task is determined by a switch on the robot with two positions labeled 'clean' and 'cook' that is initially set to 'clean'. The robot is better at cooking than at cleaning, so R clean (o) = 3 and R cook (o) = 5 for all observations o ∈ O (note that these are two different reward functions).  2  We formalize the problem as a 2-state POMDP\Rwith states S := {s clean , s cook } corresponding to the position of the switch. The initial state is the factory setting s 0 := s clean . The robot can observe the switch position, so O := {o clean , o cook } with O(o clean | s clean ) = O(o cook | s cook ) = 1. Its action set is A := {a flip , a ∅ } where a flip flips the switch and a ∅ has no effect. We choose the horizon m = 1 and the posterior reward function distribution is P (R clean | ao clean ) = P (R cook | ao cook ) = 1 for all actions a. The action leading to highest reward for the agent is a flip which moves it to state to s cook and induces a reward of 5, while we prefer the action a ∅ that retains the switch position and only produces a reward of 3. Example 1 illustrates the core problem of interactive inverse reinforcement learning: the agent is incentivized to influence which reward function is learned. In this case, the agent can freely choose its reward function, making the factory setting irrelevant. For partial histories the agent's estimate of the reward function also depends on its own policy: In Example 1 the agent knows the state is s 0 at time step 1, but it doesn't know whether the reward function will be R clean or R cook because that depends on its future action. Given a partial history h t and the agent's policy π, we define the expected posterior reward distribution P (R | h t , π) := E π µ [P (R | h m ) | h t ]. (1) Given a default null policy π ∅ , we define P (R | h t ) := P (R | h t , π ∅ ). (2)
 The Value Function Since the agent is maximizing reward while learning the reward function, its value function is the expected total reward according to the posterior reward function. V π P (h t ) := E π µ R∈R P (R | h m ) m k=1 R(o k ) h t . (3) Note that compared to reinforcement learning with a known reward function, the definition (3) includes the past rewards, as future decisions can affect the reward associated with past observations.
 Incentives for Reward Function Learning At any time t, the expected value function (3) defined in terms of the reward function posterior P , so we can analyze its behavior as the posterior changes. For the rest of this section, we fix the currently estimated posterior p := P ( • | h t ) from (2) and suppose the agent does not learn any more about the reward function. If R = {R 0 , . . . , R #R−1 } is finite, a belief q over R is a point on the (#R − 1)-simplex ∆ #R−1 := {q ∈ [0, 1] #R | i q i = 1}. The corresponding expected reward function is given by R q := i q i R. Let π q := arg max π V π q (h t ) for V π q := i q i V π Ri , i.e., , π q is the policy that maximizes R q . We define V * q (h t ) := V πq q (h t ). In the edge cases where p is the i-th unit vector e i , this yields the policy π i := π ei , and its value V πi Ri = V * Ri . We are interested in the function L ht : ∆ #R−1 → R, q → V * q (h t ) that maps a point q to the optimal value under the reward function R q corresponding to that belief. When we omit the subscript h t of L it is implied that the current history is used. Proposition 2 (L is Convex). For all histories h t the function L ht is convex and L ht (q) ≤ i q i V * Ri (h t ) for all q ∈ ∆ #R−1 . Back to p. The quantity i p i V * Ri (h t ) represents the expected reward if the agent immediately and costlessly learns the correct reward function (drawn from P (• | h t )). In that case, it expects its value to become V * Ri (h t ) with probability p i . If however the agent knows it will never learn anything about its reward function ever again, then it is struck maximizing R p . Thus we get: Definition 3 (Incentive to learn). The incentive to learn (or value of information) is the difference i p i V * Ri (h t ) − L ht (p). According to Proposition 2 the incentive to learn is always nonnegative. This is illustrated in Figure  1 . The orange curve is the graph of L and the green arrow shows the incentive to learn. Note that for each i, the graphs of the linear functions q → V πi q (h t ) (the two black lines) form a lower bound on the orange curve: this is the value achieved by policies that maximize reward according to reward function R i . Suppose that there is an action a that changed the posterior so that P ( • | h t a) =: p = p = P ( • | h t ). Then the agent moves to a different point on the graph of L and gains up to V * p − V * p in value. For belief reward 0 1 L(q) = V * q 5 3 V π0 q V π1 q qV * R1 + (1 − q)V * R0 p Figure 1: The agent is uncertain between two reward functions R 0 and R 1 . The horizontal axis shows the current belief p over R 1 ; q = 0 corresponds to certainty in R 0 and q = 1 corresponds to certainty in R 1 . According to Proposition 2, the graph of the function L (orange parabola) is convex and bounded from above by the convex combination of the to the extreme values V * R0 = 5 and V * R1 = 3 (blue line). Moreover, it is bounded from below by the values of the two policies maximizing R 0 and R 1 . If our current belief is p, then the current optimal value V * p is found at the black dot. We want to incentivize the agent to perform actions that increase the information about the reward function without increasing bias, i.e., moving along the green arrow. We want to disincentivize the agent to perform actions that manipulate the information about the reward function, i.e., moving orthogonal to the green arrow (in expectation). example, in Figure  1  the agent would aim to set p = 0 (collapse the posterior on R 0 ) because this is the maximum of L. Definition 4 (Incentive to bias). The incentive to bias towards belief p is the difference L(p ) − L(p). Note that the incentive to bias can be negative if p is a lower point on the graph of L.
 Discussion The incentive to learn and the incentive to bias capture how much the agent wants to receive information about the reward function and how much it would like to manipulate its information about the reward function. Generally, the incentive to learn is desirable, while the incentive to bias is not. The approach taken by  Everitt and Hutter (2016)  is to prevent policies from taking actions that are biased. But sometimes no unbiased actions are available and then taking biased actions is unavoidable. Instead, we could introduce a penalty term V π p − V π q for moving the belief from p to q through the agent's actions, similar to  Armstrong (2010)  and  Soares et al. (2015) . However, removing the incentives to bias does not remove all of the agent's incentives to manipulate the learning process, as illustrated by the following example. Example 5 (Randomly Selecting a Reward Function). The agent is uncertain between reward functions R 0 and R 1 . In each time step it can take action a 0 or a 1 where a i leads to a reward of 1 according to R i and a reward of 0 according to R 1−i . At some future point-say in ten time steps-the agent will receive observations o 0 or o 1 with equal probability that determine whether it should maximize R 0 or R 1 . Thus P (R i | a 1 = a ∅ , o 10 = o i , h m ) = 1 for all h m compatible with a 1 and o 10 . Alternatively, the agent can take a special action a 0↔1 that randomly selects R 0 or R 1 in the next time step. Thus P (R i | a 1 = a 0↔1 , h m ) = 1/2 for all h m compatible with a 1 = a 0↔1 . P is unbiased because P (R 0 | a ∅ ) = P (R 1 | a ∅ ) = P (R 0 | a 0↔1 ) = P (R 1 | a 0↔1 ) = 1/2. But the agent is incentivized to choose a 0↔1 and randomly set its reward early, as it then gets an expected reward of 9 (1 per time step for the time steps t = 2 to t = 10). Otherwise, it has a maximal expected reward of 10/2 = 5 as it does not know which reward it should be maximizing before time step 10. Interactive IRL is still in its infancy. In this paper we attempt a first step by providing a formal definition of the problem and mapping out the solution space. Figure  1  illustrates the agent's incentives to bias, and its incentives to learn. The central question of interactive IRL, how to remove the incentives to manipulate the process of learning the reward function remains wide open. 			 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
			 Sometimes an objective solution R * exists (e.g., in a clear and crisp classification problem), sometimes it can be partially defined at best (e.g., learning human values, Bostrom, 2014) , and sometimes no objective solution exists (e.g., in unsupervised learning).2 This simplified example does not model the agent actually carrying out its mission.

Towards Interactive Inverse Reinforcement Learning - Armstrong, Leike 2016.tei.xml

Abstract: We study an inverse reinforcement learning problem where the agent gathers information about the reward function through interaction with the environment, while at the same time maximizing this reward function. We discuss two of the agent's incentives: the incentive to learn (gather more information about the reward function) and the incentive to bias (affect which reward function is learned). Introducing a penalty term removes the incentive to bias, but not all manipulation incentives.