Introduction Would you play a game of Russian roulette? Would you take a six chamber revolver, put one bullet in, give it a spin, point it at your head, and pull the trigger? How about for a million dollars? Would you play? I would guess that most readers of this paper would not play. I would guess that you would think that a chance of one million dollars is not worth it to take this risk of ending up with a bullet in the brain. I personally would not play, for the same reason. Our brains and our lives are simply worth more than that. But suppose your life circumstances were different. Suppose you were struggling with money, that you were basically broke. Suppose you were sick, with a chronic condition you cannot afford to cure. Maybe you do not have that many years left to live anyway. Now, with less to lose and more to gain, that game of Russian roulette might start to look more attractive. Now, you might start counting how much that million dollars could do for you. Could it cure your sickness, make you healthy again? Could it add years to your life? Could it pull you out of poverty? Could it give you basic comfort? If the million dollars would do enough for you, then maybe you would choose to play. Desperate circumstances can sometimes warrant taking desperate risks. If it works, the circumstances get better, maybe much better. But it might not work, and if it does not, it comes with a downside-in this case, a bullet in the brain. Whether to play the game is a downside dilemma: a dilemma involving a significant possible downside. This paper talks of a great downside dilemma. It is great because the stakes are so high-indeed, they are literally astronomical. At stake is not the fate of a single person, as in Russian roulette, but the fate of human civilization. This includes the roughly seven billion people alive today and the many more members of all the future generations that might ever live. The stakes are astronomical because humans (or our descendants) might be able to colonize space and achieve great things across the Universe. Human civilization already has an active space program, and space colonization seems feasible, as long as no great catastrophe denies humanity the chance. The rest of the Universe is vastly larger than our humble home planet, so space colonization would open up enormous new opportunities. Meanwhile, for all humanity currently knows, humans might have the only intelligent civilization anywhere in the Universe. And so the stakes could mean nothing less than the success or failure of intelligent civilization in the entire universe. A great downside dilemma, indeed. | Royal Swedish To be more specific, the great downside dilemma is any circumstance in which human civilization must choose whether to take a risk in which, if it works out, the benefit greatly improves the human condition, but if it does not work out, a catastrophe will occur, a catastrophe so large that civilization could perish, a metaphorical bullet in the brain. The dilemma is whether to take the risk. How much does civilization value that improvement in its condition? Could it be enough to pull civilization out of desperate circumstances? How large is the risk of catastrophe? Is it small enough that the risk is worth taking? Can any risk of civilization perishing be small enough to justify taking the risk? These questions must be answered in order to decide whether to take the risk. The great downside dilemma arises often for decisions about whether to pursue certain emerging technologies. These technologies promise to solve major societal problems. They bring peace, cure disease, protect the environment, and more. Or rather, they do these things if they work as intended. However, they may not work out as intended. They may fail, and fail catastrophically. In the worst cases, they could kill every living human-the extinction of our species-and destroy much of the rest of Earth's biosphere as well. Should society develop and launch these technologies, given their promise and despite their risks? That is the great downside dilemma for emerging technologies. This dilemma is an important issue for society as a whole and especially for scientists and engineers, who by virtue of their background are especially able to contribute to the debate. This dilemma is one important part of the broader challenge of avoiding civilization-ending global catastrophes. A growing body of scholarship recognizes the avoidance of these catastrophes as crucial for the long-term success of human civilization, and likewise as a key priority for action today  [1] [2] [3] [4] [5] [6] [7] [8] . Visionary technologist James Martin likened this era of civilization to a turbulent river that it must navigate  [9] . If this era of civilization successfully navigates the river, then a long, bright future awaits, both on Earth and beyond. However, if it fails, then human civilization suffers a premature death. This paper describes several great downside dilemmas for emerging technologies and explains how humanity can navigate through them. The paper also discusses some other technologies that do not pose this dilemma because they promise to bring major benefits without a significant catastrophic risk.
 Historical precedents Amazingly, the great downside dilemma for emerging technologies has been faced at least twice before. The first precedent came in the desperate circumstances of World War II. The dilemma was whether to test-detonate the first nuclear weapon. While nuclear weapons proved to be unprecedentedly destructive weapons, a single detonation did not destroy the entire planet as some initially feared. The second precedent came during calm circumstances but still posed a dilemma every bit as large: whether to engage in messaging to extraterrestrial intelligence (METI). METI is of note because the dilemma still has not been resolved. Humanity still does not know if METI is safe. Thus METI decisions today face the same basic dilemma as the initial decisions in decades past.
 Nuclear weapons It was 1945, towards the end of World War II. An American team of physicists, engineers, and military personnel built the first atomic bomb, which they named Trinity. Trinity was to be detonated in a test explosion, to make sure the technology worked, before using additional atomic bombs against Japan. By that point, Germany had already surrendered. Japan was nearing defeat, and the United States believed that the atomic bomb could compel Japan to surrender without the US waging a long, bloody invasion. It might seem counterintuitive, but this most powerful of weapons was built to save live  1  . However, some of the physicists worried that the test might fail catastrophically. They worried that the detonation could ignite the atmosphere, ending life on Earth. They believed the chance of this happening to be exceptionally small, due to their understanding of the relevant physics. Still, they closed their report on the topic with the line 'However, the complexity of the argument and the absence of satisfactory experimental foundations makes further work on the subject highly desirable'  [11] . Thus the risk did give them some pause. Sure enough, they took the risk. As is now known, the Trinity test succeeded: the bomb worked, and the atmosphere did not ignite. The rest is history. Humanity survived the first atomic bomb detonation, the next two, which were dropped on Hiroshima and Nagasaki,  1  Reed  [10]  reviews the history of the atomic bomb development and the corresponding physics. and the 2054 atomic bombs that have been detonated since in further testing  2  . The largest of these bombs, the Soviet Tsar Bomba, had a yield equivalent to about 50 megatons of TNT, a whopping 2500 times larger than Trinity. The atmosphere did not ignite. And physics has by now progressed to the point where we understand with very high confidence why atomic bomb detonations would not cause these harms (though they can of course cause other harms). But for that brief moment in time, when the first atomic detonation was under consideration, a great downside dilemma was faced, without the benefit of hindsight that now exists. Today, nuclear weapons remain a major threat. A single nuclear weapon could kill thousands or even millions of people. Nuclear war with hundreds or thousands of weapons (about 17 000 weapons still exist, mainly held by the United States and Russia) would not produce enough radiation to cause human extinction, as Bertrand Russell, Albert Einstein, and others once feared  [12] . But it would cause significant cooling, as the smoke from burning cities rises into the atmosphere and blocks incoming sunlight. This cooling, often known as nuclear winter, could cause widespread agricultural failure, with the resulting famine threatening millions or even billions of people  [13, 14] . The worst case scenario could include human extinction  [15] . The leaders of nuclear weapons states thus face a different dilemma: In a crisis, are nuclear weapons worth using? While nuclear weapons are no longer a new technology, large nuclear arsenals have never been used in war, and so the dilemma must still be resolved without the benefit of hindsight.
 Messaging to extraterrestrials It was 1974, a relatively calm and ordinary year by most measures. But an unusual exercise was in preparation at an astronomy observatory in Arecibo, Puerto Rico. The Arecibo Observatory hosted what was (and still is) the largest radio telescope in the world  3  . Usually, the telescope is used either for radio astronomy, which detects radio waves incoming from the rest of the Universe, or radar astronomy, which studies the solar system by sending radio waves towards planets and other nearby objects and analyzing the waves that bounce back. Radio and radar astronomy are generally harmless and of scientific value. But in 1974, the Arecibo telescope was to be used differently. The plan was to send a message from Arecibo to a cluster of stars 25 000 light years away. The Arecibo Message was designed by astronomer Frank Drake and colleagues with the premise of METI. The message contained seven parts describing human physiology and astronomy. This was not the first exercise in METI. In 1962, the Morse Message was sent from the Evpatoria Planetary Radar in Crimea to Venus. But the Morse Message was as harmless as regular radar astronomy studying Venus. Because the Arecibo message was broadcast elsewhere, it broke new ground. So far, the Arecibo message has not received a response. Of course it has not: it was sent 40 years ago to a location 25 000 light years away. It will take at least another 49 960 years for the message to arrive and the response to reach back to Earth 4 . And it is possible that no ETI will receive the Arecibo message. Indeed, it is possible that there are no ETI out there to receive it. It is also possible that ETI will receive it but not respond in any way. So even after another 49 960 years, the Arecibo message could prove inconsequential to humanity, except for its modest educational value. But it might not. The message could receive a response. Despite some proclamations to the contrary, humanity has little understanding of how an encounter with ETI would proceed. Some people expect that ETI would benefit humanity, providing scientific knowledge and intercultural exchange, or even solutions to humanity's problems. Others expect that ETI would harm humanity, enslaving or even killing us. These are among the many possible outcomes of ETI encounter  [16, 17] . Presumably, if it was known that the outcome would be beneficial, METI would proceed; likewise it would not proceed if the outcome was known to be harmful  [18] . But in 1974, it was not known. Whether to engage in METI thus posed a great downside dilemma. In 2014, at the time of this writing, it is still not known whether METI is safe. Since the Arecibo message, several other messages to ETI have since been sent to closer stars, most recently the 2013 Lone Signal project  5  . None of these messages has yet received a reply. Meanwhile, it is an exciting time for SETI-the search for ETI. Astronomers are just starting to discover extrasolar planets  [19] . But no ETI have yet been found. Until then, humanity will have deep uncertainty about the merits of METI. Some progress can be made by carefully thinking through the possibilities of ETI encounter. An argument can be made that no high-power METI should be conducted until humanity better understands the risks  [20] , but this is a controversial point. The great downside dilemma for METI persists.
 Dilemmas in the making While humanity continues to face dilemmas related to nuclear weapons and METI, new dilemmas lurk on the horizon. The stakes for these new dilemmas are even higher, because they come with much higher probabilities of catastrophe 6 . I will  2  Atomic bomb detonation data can be obtained from the Comprehensive Nuclear-Test-Ban Treaty Organization at http://www.ctbto.org/nucleartesting/history-of-nuclear-testing/world-overview/page-1-world-overview/.  3  Arecibo is the world's largest single-aperture radio telescope. Arrays of multiple telescopes combined as astronomical interferometers collectively cover larger areas.  4  Assuming the location is exactly 25 000 light years away, then 49 960 years from now is the minimum time it could take for a response to reach Earth. The response could reach Earth later if the ETI take more time before transmitting the response.  5  Full disclosure: I received funds from Lone Signal to contribute to a risk analysis of Lone Signal's transmissions. The study concluded that the transmissions posed no significant risk because the transmitter Lone Signal was using at the time did not exceed the background radio wave leakage from radio and television broadcasts  [18] .  6  This assumes that the probability of catastrophe from METI is relatively low. This is a debatable point, given the deep uncertainty surrounding the possibility of extraterrestrial contact. focus on two: stratospheric geoengineering and artificial general intelligence. Neither technology currently exists, but both are subjects of active research and development. Understanding these technologies and the dilemmas they pose is already important and will only get more important as the technologies progress.
 Stratospheric geoengineering In summer 2010, heavy monsoons flooded about one fifth of Pakistan, with millions of people affected  [21] . The floods were part of a broader northern hemisphere summer heat wave that set temperature records in many locations. Vast wildfires in Western Russia produced so much smoke that people in Moscow wore masks and airports redirected traffic. The floods, heat wave, and wildfires are among the sorts of extreme weather events that are expected to happen more often and with greater intensity as global warming worsens  [22]    7  . The standard means of lessening the harms of global warming is to reduce atmospheric emissions of carbon dioxide, methane, and other greenhouse gases. That means using energy more efficiently, switching away from coal power, reversing deforestation, and more. But despite the risks of global warming, greenhouse gas emissions have been steadily increasing, and are projected to continue increasing into the future. More emissions means warmer temperatures and more severe harms from extreme weather events, sea level rise, and more. In despair over the perceived failure to reduce emissions, observers are increasingly considering geoengineering to lower global temperatures. Geoengineering is the intentional manipulation of the global Earth system  [27] . Greenhouse gas emissions do not qualify as geoengineering because they are an unintended byproduct of activities with other aims. Perhaps the most commonly discussed type of geoengineering is stratospheric geoengineering, which would lower temperatures by injecting particles into the stratosphere, thereby blocking a portion of incoming sunlight. Stratospheric geoengineering is attractive because of its relative feasibility, efficacy, and affordability. However, stratospheric geoengineering changes regional temperature and precipitation patterns, leaving some need to adapt to climatic changes. Stratospheric geoengineering also does nothing to address the acidification of oceans caused by carbon dioxide emissions being absorbed into oceans. Ocean acidification is a major problem in its own right, a large threat to ocean ecosystems. Finally, stratospheric geoengineering also poses significant risks that could even exceed those of global warming. Perhaps the largest risk from stratospheric geoengineering is the possibility of abrupt halt. Particles put into the stratosphere will cycle out on time-scales of about five or ten years. In order to maintain stable temperatures, particles must be continuously injected into the stratosphere. If the geoengineering abruptly halts, such that additional particles are not injected, then temperatures will rapidly shoot back up towards where they would have been without geoengineering  [28] . This rapid temperature increase could be especially difficult to adapt to and thus be especially disruptive. For example, it may be difficult to determine which crops to plant in a given region, because the crops suitable for that region will change too quickly. Fortunately, the rapid temperature increase can be avoided simply by continuing to inject particles into the stratosphere. Indeed, the harms of rapid temperature increase provide strong incentive to not to stop particle injection in the first place. Under normal circumstances, people would have to be either incompetent or malicious to stop particle injection. Assuming the geoengineering is managed by responsible parties, abrupt halt may be unlikely, making stratospheric geoengineering relatively safe. This is under normal circumstances. However, particle injection may nonetheless halt if some other catastrophe occurs, such as a war or a pandemic, that prevents people from continuing the injections. The result would be a 'double catastrophe' of rapid temperature increase hitting a population already vulnerable from the first catastrophe  [29] . This double catastrophe could be very harmful; potentially it could even result in human extinction. This makes for a rather severe downside. Figure  1  depicts the double catastrophe scenario. The figure shows average global temperature versus time for three scenarios: (1) the world without geoengineering, in which temperatures gradually rise due to greenhouse gas emissions; (2) ongoing geoengineering, in which temperatures remain indefinitely at a low, stable level (around 13 °C); and (3) abrupt geoengineering halt (around the year 2080), in which temperatures rapidly rise towards where they would have been without geoengineering. Figure  1  also indicates when an initial catastrophe would occur (shortly before 2080) in a double catastrophe scenario. The great downside dilemma for stratospheric geoengineering is the dilemma of whether to inject particles into the Figure  1 . global temperature for three scenarios: no geoengineering, ongoing geoengineering, and geoengineering that abruptly stops. The initial catastrophe corresponds to a geoengineering double catastrophe  [29] .  7  I am using the term 'global warming' here instead of the usual 'climate change' to distinguish it from nuclear winter, which is also a climatic change. Any readers who still doubt the legitimacy of global warming as an issue should consult some of the many works on the topic, including accessible books by leading global warming researchers  [23, 24]  and my own humble contribution  [25] . Global warming research is also voluminously reviewed by the Intergovernmental Panel on Climate Change  [26] . stratosphere. On one hand, stratospheric geoengineering could lower temperatures, avoiding many harms of global warming. On the other hand, it poses a risk of rapid temperature increase that could even result in human extinction. So, should stratospheric geoengineering be pursued? A key factor in resolving the dilemma is understanding how bad the impacts of global warming could get without geoengineering. The floods, heat waves, and other effects already being observed will almost certainly get worse. This is bad, but there is an even worse impact potentially on the horizon: the exceedance of mammalian thermal limits. The limits depend on wet bulb temperature, which is a combination of 'regular' dry bulb temperature and humidity. When wet bulb temperature goes above 35 °C, mammals-including humans-can no longer perspire to regulate our body temperature, causing us to overheat and die. Currently, wet bulb temperatures never exceed the 35 °C limit, but under some possible global warming scenarios, the limit would sometimes be exceeded in much of the land surface of the planet  [30] . Unless humans and other mammals took shelter in air conditioning, they would die. Under these conditions, it may be difficult to keep civilization intact in the warmer regions or even worldwide. Another perspective on the potential severity of global warming comes from looking at the long-term co-evolution of the human species and Earth climates. The species Homo sapiens sapiens is dated at around 200 000 years old. This means that humans have lived through about two full glacialinterglacial cycles, i.e. ice ages and the warm periods between them, which cycle back and forth on time-scales of about 100 000 years  [23] . Archaeological evidence suggests that early Homo sapiens sapiens and their immediate ancestors had cognitive capabilities comparable to those of contemporary humans  [31, 32] . However, civilization did not take off until the agricultural revolution, which began around 10 000 years ago and occurred in at least seven independent locations within just a few thousand years. The last 10 000 years coincide with the Holocene, a warm interglacial period with a relatively stable climate, suggesting that this climate may have been crucial for the rise of civilization  [33] . Meanwhile, global warming threatens to push temperatures to levels significantly outside the range of recent glacial-interglacial cycles, bringing climates that Homo sapiens sapiens and its immediate ancestors have never seen before  [34] . To the extent that certain climates are essential for human civilization, global warming could be devastating. This sort of long-term perspective is also helpful for understanding stratospheric geoengineering risk. Global warming could last for centuries, millennia, or even longer  [34] . This is a very long time to continue injecting particles into the stratosphere. It is also plenty of time for plenty of catastrophes to occur. For example, risk analysis of nuclear war finds about a 0.1%-1% chance of nuclear war occurring during any given year  [35] . Over hundreds or thousands of years, this makes nuclear war virtually certain to occur. Of course, the world could permanently get rid of nuclear weapons, eliminating the risk. But this might not happen, and meanwhile there are other types of catastrophes to worry about. Over the time-scales of global warming, a stratospheric geoengineering double catastrophe may be quite likely. So, should stratospheric geoengineering be pursued? At this time, I do not believe a good answer to this question exists. There is too much uncertainty about both the consequences of stratospheric geoengineering and the consequences of not stratospheric geoengineering, as well as the probabilities of stratospheric geoengineering abrupt halt. Fortunately, the decision does not need to be made just yet. Global warming is not yet so bad that stratospheric geoengineering is worth the risk. But geoengineering decisions may be made soon; research to reduce the uncertainty should proceed now so that wise decisions can be made  [36, 37] . When the time comes to decide whether to launch stratospheric geoengineering, the right action to take may also be the more difficult action to take. It is quite plausible that civilization could endure the worst harms of regular global warming, but would collapse from the rapid global warming of stratospheric geoengineering abrupt halt. If this is the case, then it would be in civilization's long-term interest to abstain from stratospheric geoengineering and suffer through regular global warming. This abstention may be best regardless of how painful regular global warming could get and regardless of how unlikely stratospheric geoengineering abrupt halt would be. It is just that important to ensure the long-term viability of human civilization. But this is hardly an encouraging prospect, dooming humanity to the pains of regular global warming, when lower temperatures could so easily be produced. Meanwhile, civilization can lessen the stratospheric geoengineering dilemma by reducing greenhouse gas emissions. Regardless of any broader failures at emissions reductions, every additional bit helps. And reducing emissions helps with both sides of the dilemma: it lessens the severity of both regular global warming and the rapid temperature increase from stratospheric geoengineering abrupt halt. As discussed further below, many options for reducing emissions come with minimal dilemmas of their own, making them excellent options to pursue.
 Artificial general intelligence In January 2011, two world-leading players of the game show Jeopardy! took on an IBM computer named Watson. Watson won the game convincingly. During the last Final Jeopardy! round, human contestant Ken Jennings wrote below his response question, 'I for one welcome our new computer overlords'. It was a humorous moment in the long rise of artificial intelligence (AI). But how high can AI rise? Could AI actually become the overlords of humanity, taking over the world? And should such a development be welcomed? At the outset, it is important to distinguish between two types of AI: narrow and general. Narrow AI is intelligent in specific domains but cannot reason outside the domains it was designed for. Narrow AI is by now ubiquitous across a breadth of contexts, from searching the web to playing games like Jeopardy! narrow AI can be quite useful, and can also pose some risks. But it is not expected to take over the world, because controlling the world requires capabilities across many domains. General AI (AGI) is intelligent across a wide range of domains. Humans are also intelligent across many domains, but this does not mean that AGI would necessarily think like humans do. An AGI may not need to think like humans in order to be capable across many domains  8  . Early AI researchers boldly predicted that human-level AGI would be achieved by dates long since past, as Crevier  [39]  and McCorduck  [40]  chronicle. This grandiose failure of prediction led many modern AI researchers to be skeptical about the prospects of AGI  [41] . However, there remains an active AGI research community  [42] . Experts in the field diverge widely about when AGI is likely to be achieved and on its impacts if or when it is achieved  [43] [44] [45] . But some of the predictions about impacts are quite dramatic. One line of thinking posits that an AGI, or at least certain types of AGIs, could essentially take over the world. This claim depends on two others. First, power benefits from intelligence, such that the most intelligent entities will tend to have the most power. Second, an AGI can gain vastly more intelligence than humans can, especially if the AGI can design an even more intelligent AGI, which designs a still more intelligent AGI, and so on until an 'intelligence explosion'  [46]  or 'singularity'  [47, 48]  occurs. The resulting 'superintelligent' AGI  [49]  could be humanity's final invention  [50]  because the AGI would then be fully in control. If the AGI is 'Friendly' to humanity  [51] , then it potentially could solve a great many of humanity's problems. Otherwise, the AGI will likely kill everyone inadvertently as it pursues whatever goals it happened to be programmed with-for example, an AGI programmed to excel at chess would kill everyone while converting the planet into a computer that enabled it to calculate better chess moves  [52] . Per this line of thinking, an AGI would be much like a magic genie, such as the one depicted in the film Aladdin (John Musker and Ron Clements, directors, 1992). The genie is all-powerful but obligated to serve its master. The master can wish for almost anything, but should be careful what he or she wishes for. Indeed, genie stories are often stories of unintended consequences. For example, in the penultimate scene of Aladdin, Jafar wishes to become a genie. He was eager to gain the genie's powers, but ended up trapped in servitude (and stuck inside a small lamp). The story with AGI may be similar. The AGI would do exactly what its human programmers instructed it to do, regardless of whether the programmers would, in retrospect, actually want this to happen. In attempting to program the AGI to do something desirable, the programmers could end up dead, along with everyone else on the planet  9  . If this line of thinking is correct, or even if it has at least some chance of being correct, then AGI poses a great downside dilemma. Should an AGI be built and launched? Given the possibility of being destroyed by AGI, it might appear that AGI should simply not be built. Doing so would ensure that humanity retains control of itself and its fate. But for several reasons, the situation is not so simple. A first complication is that AGI might be Friendly or otherwise beneficial to humanity, or to the world. The benefits of a Friendly AGI could be immense. Imagine having the perfect genie: unlimited wishes that are interpreted as you intended them to be, or maybe even better than you intended them to be. That could go quite well. Perhaps there would be no more poverty or pollution. Perhaps space colonization could proceed apace. Perhaps the human population could double, or triple, or grow tens, hundreds, or thousands of times larger, all with no decline in quality of life. A Friendly AGI might be able to make these things possible. Decision-making on AGI should balance this large potential upside with the also-large downside risk. For example, suppose the AGI had a 50% chance of killing everyone and a 50% chance of doubling the human population with no decline in quality of life. The expected population would be equally large with or without the AGI. Does this mean that humanity is indifferent to launching the AGI? If it was a 51% chance of doubling the population, versus 49% for killing everyone, does this mean humanity would rather launch the AGI? What if it was a chance of the population increasing by a factor of ten, or a thousand? These are important types of questions to answer when making decisions about launching an AGI. A second complication is that AGI is not the only threat that humanity faces. In the absence of AGI, humanity might die out anyway because of nuclear weapons, global warming, or something else. If AGI succeeds, then these other threats could go away, solved by our new computer overlords. That is a significant upside for AGI. What if an AGI has a 50% chance of killing everyone, but absent AGI, humanity has a 60% of dying out from something else? Should the AGI be launched? The dilemma for AGI can thus look a lot like that for stratospheric geoengineering. Imagine, some years into the future, humanity finds itself in a difficult spot. Perhaps global warming is bringing great harm, and other environmental stressors are as well. Perhaps major countries are on the brink of war with nuclear weapons or something even more destructive. Perhaps poverty is rampant, life unsafe and unpleasant. Perhaps other solutions, including stratospheric geoengineering, are found to be unsafe or otherwise undesirable. And perhaps there is no hope in sight of conditions improving. In this case, taking the risk of launching a AGI could start to look attractive. Indeed, in terms of the long-term success of human civilization, it might even be the right thing to do. Or, the right thing may be to suffer through without AGI. It would depend on the details, just as it would for stratospheric geoengineering or a desperate game of Russian roulette. Following this logic, one way to help reduce AGI risk is to improve the general human condition. By keeping humanity out of desperate circumstances, the risk of AGI can be made to look less attractive. This opens up a wide range of opportunities to help reduce AGI risk, from reducing  8  AGI was discussed in detail in another paper in the series of papers based on the event 'Emerging Technologies and the Future of Humanity'  [38] .  9  Arguably, such a result would at least be better than an eternity trapped in a small lamp. greenhouse gas emissions to improving conditions for the world's poor. But the merits of this approach depend on how the AGI would be developed. The third complication is that AGI development could involve basic computing resources and technologies of growing economic importance. AGI is not like nuclear weapons, which require exotic materials. AGI could be developed on any sufficiently powerful computer. Computing power is steadily growing, a trend known as Moore's Law. Meanwhile, narrow AI is of increasing technological sophistication and economic importance. At the time of this writing, driverless cars are just starting to hit the streets around the world, with promise to grow into a major industry. Differences between narrow AI and AGI notwithstanding, these AI advances may be able to facilitate the development of AGI. Given the risks of AGI, it may seem attractive or even wise to relinquish precursor hardware and software technologies, potentially including certain narrow AI and the computer systems they run on  [53] . But given the pervasiveness of these technologies, it may be difficult to do so. Here lies another dilemma. Would humanity be willing to sacrifice much of its computing technology in order to avoid an AGI catastrophe? Should it? The dilemma here resembles that faced in the recent film Transcendence (Wally Pfister, director, 2014). The film shows an AGI that has been launched and is steadily taking over the world. The AGI is in many ways beneficial or even friendly, but the humans who are close to it become increasingly skeptical and decide to shut it down. (More precisely, they persuade the AGI to shut itself down, since the AGI was still in control.) However, in shutting it down, humanity had to sacrifice the internet, and potentially also other electronics. A case can be made that the AGI should not have been shut down: without the internet and other electronics, the long-term prospects for human civilization could be severely limited, such that humanity would be better off keeping the AGI intact and hoping for the best  [54] . As with stratospheric geoengineering, AGI launch decisions do not need to be made right now. However, for AGI there is great uncertainty about how much time remains. Experts are sharply divided on how long it will take to achieve AGI, with some doubting that it will ever occur. Given this uncertainty, and the high stakes of the launch decision, it is not at all too early to assess which AGIs should or should not be launched, and to create the conditions that can help ensure better outcomes whether or not an AGI is launched.
 Technologies without great downside dilemma Not all technologies present a great downside dilemma. These technologies may be disruptive, may have downsides, and may carry risks, but they do not threaten catastrophic harm to human civilization. Or, to the extent that they could threaten catastrophic harm, they do not increase the risk of catastrophe beyond what it would be without the technology, or do not increase the risk to any significant extent. Some of these technologies even hold great potential to improve the human condition, including by reducing other catastrophic risks. These latter technologies are especially attractive and in general should be pursued to the extent that their benefits and cost-effectiveness are competitive with other options for improving the human condition (and achieving any other goals). Three such emerging technologies are discussed here.
 Sustainable design Sustainable design refers broadly to the design of technologies oriented towards improving the environment, advancing sustainability, and related goals. These technologies promise to reduce the harms of climate change and other environmental problems. Quite a lot of such technologies are already in use, from the humble bicycle to advanced solar technologies. This is a vast technology space, and a lot has been said about these technologies elsewhere  [55] , so a full review here is unwarranted. What is worth noting here is that these technologies can reduce the risk of environmental catastrophes like climate change, and are often also worth pursuing for other reasons. For example, technology that uses energy, water, and other resources more efficiently can save money by avoiding purchases of these resources. Technologies like bicycles can make people healthier by giving them more exercise. Where sustainable design comes with such cobenefits, it is an especially attractive option. But given the catastrophic potential of environmental risks, some sustainable design, and potentially quite a lot of it, is worth pursuing even if it otherwise comes at an expense.
 Nuclear fusion power Nuclear fusion power is perhaps the Holy Grail of sustainable design. It promises a clean, safe, abundant energy source. If nuclear fusion power can be realized, and if it can be made affordable, then humanity's energy needs could potentially be fully met. And with abundant energy, a lot of other opportunities open up. For example: ocean water could be desalinated, eliminating water resource scarcities. Carbon dioxide could be removed from the atmosphere, which is another form of geoengineering, and a much safer one at that. Countries can develop their economies without worrying nearly as much about their environmental impact and without worrying about being dependent on another country's energy resources. One major long-term benefit of fusion power relates back to the fossil fuels it would replace. With fusion power, humanity can keep the rest of the fossil fuels underground, ready and waiting for when they will really be needed. That time will come sometime within the upcoming hundreds of thousands of years, when Earth's climate cycles back to a glacial period: a new ice age. The exact timing of the next glacial period is uncertain, and depends on, among other things, how much greenhouse gas humanity emits  [34] . But, barring any other radical changes to the global Earth system (such as its dismantling by a runaway AGI), a glacial period will eventually occur. And when it does, it could help to still have some fossil fuel around to lessen the bite of the global cooling  [56, p 234-235] . This would be yet another form of geoengineering, one with the long-term interests of human civilization in mind. Unfortunately, it is not clear if or when the Holy Grail of fusion power will be achieved. Fusion power research has been going on for decades  [57] , and it may take more decades still. With such a long development period, fusion power is a modern analog to cathedrals. Many cathedrals took a century or longer to build. This includes at least one cathedral currently under construction, Sagrada Família in Barcelona, whose construction began in 1882 and has no clear projected completion date. Humanity's track record with cathedrals indicates its capability to complete large, multi-century, intergenerational projects. Perhaps the fusion power project will be completed too. Unlike cathedrals, it is not known if it is even possible to complete the fusion power project: to make fusion power a major energy source for human civilization. Already, it is possible to generate power from nuclear fusion. First came uncontrolled fusion-fusion bombs-beginning with the detonation of Ivy Mike in 1952. Soon after came controlled fusion, beginning with Scylla 1 at Los Alamos National Laboratory in 1958  [58] . Controlled fusion is what can be used for electricity generation. However, controlled fusion thus far has always consumed more energy than it generates. A major breakthrough recently occurred at the National Ignition Facility at Lawrence Livermore National Laboratory: for the first time, a net energy gain occurred within the fuel that triggers the fusion  [59] . However, the fuel is just one part of the fusion process; the National Ignition Facility experiment consumed overall about 100 times more energy than it generates. But, clear progress is being made. On the other hand, much more progress is needed still, and it is not clear if or when net energy gain will be achieved, or if it would be affordable. If affordable fusion power is achieved, it would be transformative. The fuels are deuterium and lithium, supplies of which can last for thousands to billions of years, depending on power plant design, and there could be no significant radioactive waste  [60] . While fusion reactors potentially could be used to generate materials for nuclear weapons, their weapons proliferation risk would be lower, potentially much lower, than it is for fission power  [61, 62] . While fusion power research is expensive and the prospects for success uncertain, the potential benefits are, in my own estimation, sufficient to justify ongoing investment. This cathedral is well worth attempting to build.
 Space colonization The dream of living beyond Earth may be as old as humanity itself. Within the last century, concrete steps have been taken towards this dream. The project of colonizing space may take even longer than the project of fusion power, perhaps orders of magnitude longer. But it comes with its own set of sizable benefits, with relatively little risk. One benefit is the emotional inspiration that humanity can draw from marveling at its cosmic achievement  [63] . Other notable benefits are more practical, but no less great. One major benefit of space colonization is the protection it offers against global catastrophes on Earth. If humanity has self-sufficient space colonies, then it can survive even the complete destruction of its home planet. A spacefaring civilization is a more resilient civilization. This benefit has prompted calls for space colonization [64]. However, space colonization using current technology would be highly expensive and perhaps not even feasible, rendering other options for protecting against catastrophes, including Earthbased refuges, the more cost-effective option  [65, 66] . The protections that space colonization could offer do not justify investment in space colonization at this time. While space colonization can protect against harms, it can also enable major benefits on its own. The opportunities for civilization are, quite literally, astronomically greater beyond Earth than on it. Indeed, the astronomic potential for human civilization is a main reason why great downside dilemmas and other global catastrophic risk decisions are so important to resolve. But again, this does not mean that humanity should invest in space colonization at this time. Instead, it would be wise to focus on the catastrophic threats it faces, such that future generations can go on to colonize space and achieve astronomically great success as a civilization.
 Conclusion The fate of human civilization now hangs in the balance. As James Martin put it  [9] , humanity is going through a turbulent river full of many threats to its survival. Many of these threats derive from risky emerging technologies like stratospheric geoengineering and artificial general intelligence. Some threats also derive from established technologies like nuclear weapons and radio telescopes for messaging to extraterrestrials. And other technologies do not pose a significant threat, including sustainable design technologies, nuclear fusion power, and space colonization. Meanwhile, all of these technologies, if used properly, could help humanity navigate the turbulence. And if the turbulence is successfully navigated, a very long and bright future awaits. Humanity's future could include billions of years on Earth as well as a much bigger and longer existence across the Universe. Human civilization and its descendents can achieve many great things, if only it has the opportunity. Navigating the turbulence-preventing civilization-ending global catastrophe-is thus a crucial task for this era of human civilization. The great downside dilemma for risky emerging technologies could be an especially difficult stretch of turbulence for humanity to navigate. Technologies like stratospheric geoengineering and artificial general intelligence pose great temptations, especially if humanity finds itself in difficult circumstances. For the long-term sake of human civilization, it may be best to abstain from the technologies, but over the short-term, abstention could mean suffer through life without them. Global warming is just one of several forces that could put humanity in desperate circumstances in the not-too-distant future, making risky technologies especially attractive. If the right decisions are to be made about these various technologies-and that could mean taking the risk of using them-then two things are needed. First, the risks must be understood. People must know what the right decision is. This means characterizing the probabilities that the technologies will fail, the severity of harm if they do fail, and humanity's prospects if the technologies are not used. But, as they say, knowing is only half the battle. The other half is applying the knowledge. The second thing needed is for decision-making procedures to be in place such that bad risks are not taken. Accomplishing this means bringing together the many people involved in risky technology development, from scientists and engineers to government regulators. Some scientists and engineers might not like having their work regulated, but this only underscores the importance of including them in the process, so their concerns can be addressed, as can anyone else's. Many jurisdictions already regulate a variety of technologies, in light of the risks they pose. This is a good step. But emerging technologies pose new challenges that must be addressed in turn. And the global nature of the worst catastrophes suggests a role for international cooperation  [67] . Efforts at smaller scales can also play a role, including the daily actions everyone can make to protect the environment, promote peace, and otherwise keep humanity out of desperate circumstances. For the sake of human civilization-indeed, for the sake of the Universe-actions across all these scales are well worth taking. Academy of Sciences     Physica Scripta Phys. Scr. 89 (2014) 128004 (10pp) doi:10.1088/0031-8949/89/12/128004
			 Phys. Scr. 89 (2014) 128004 S D Baum

Abstract: Some emerging technologies promise to significantly improve the human condition, but come with a risk of failure so catastrophic that human civilization may not survive. This article discusses the great downside dilemma posed by the decision of whether or not to use these technologies. The dilemma is: use the technology, and risk the downside of catastrophic failure, or do not use the technology, and suffer through life without it. Historical precedents include the first nuclear weapon test and messaging to extraterrestrial intelligence. Contemporary examples include stratospheric geoengineering, a technology under development in response to global warming, and artificial general intelligence, a technology that could even take over the world. How the dilemma should be resolved depends on the details of each technology's downside risk and on what the human condition would otherwise be. Meanwhile, other technologies do not pose this dilemma, including sustainable design technologies, nuclear fusion power, and space colonization. Decisions on all of these technologies should be made with the long-term interests of human civilization in mind. This paper is part of a series of papers based on presentations at the Emerging Technologies and the Future of Humanity event held at the Royal Swedish Academy of Sciences on


Rodrigo Toro Icarte
Ethan Waldie
Toryn Q Klassen
Richard Valenzano
Margarita P Castro
Sheila A Mcilraith
Rodrigo Toro

Learning Reward Machines for Partially Observable Reinforcement Learning

Introduction The use of neural networks for function approximation has led to many recent advances in Reinforcement Learning (RL). Such deep RL methods have allowed agents to learn effective policies in many complex environment including board games  [30] , video games  [23] , and robotic systems  [2] . However, RL methods (including deep RL methods) often struggle when the environment is partially observable. This is because agents in such environments usually require some form of memory to learn optimal behaviour  [31] . Recent approaches for giving memory to an RL agent either rely on recurrent neural networks  [24, 15, 37, 29]  or memory-augmented neural networks  [25, 18] . In this work, we show that Reward Machines (RMs) are another useful tool for providing memory in a partially observable environment. RMs were originally conceived to provide a structured, automatabased representation of a reward function  [33, 4, 14, 39] . Exposed structure can be exploited by the Q-Learning for Reward Machines (QRM) algorithm  [33] , which simultaneously learns a separate policy for each state in the RM. QRM has been shown to outperform standard and hierarchical deep RL over a variety of discrete and continuous domains. However, QRM was only defined for fully observable environments. Furthermore, the RMs were handcrafted. In this paper, we propose a method for learning an RM directly from experience in a partially observable environment, in a manner that allows the RM to serve as memory for an RL algorithm. A requirement is that the RM learning method be given a finite set of detectors for properties that serve as the vocabulary for the RM. We characterize an objective for RM learning that allows us to formulate the task as a discrete optimization problem and propose an efficient local search approach to solve it. By simultaneously learning an RM and a policy for the environment, we are able to significantly outperform several deep RL baselines that use recurrent neural networks as memory in three partially observable domains. We also extend QRM to the case of partial observability where we see further gains when combined with our RM learning method.
 Preliminaries RL agents learn policies from experience. When the problem is fully-observable, the underlying environment model is typically assumed to be a Markov Decision Process (MDP). An MDP is a tuple M = S, A, r, p, γ , where S is a finite set of states, A is a finite set of actions, r : S × A → R is the reward function, p(s, a, s ) is the transition probability distribution, and γ is the discount factor. The agent starts not knowing what r or p are. At every time step t, the agent observes the current state s t ∈ S and executes an action a t ∈ A following a policy π(a t |s t ). As a result, the state s t changes to s t+1 ∼ p(s t+1 |s t , a t ) and the agent receives a reward signal r(s t , a t ). The goal is to learn the optimal policy π * , which maximizes the future expected discounted reward for every state in S  [32] .  [38]  is a well-known RL algorithm that uses samples of experience of the form (s t , a t , r t , s t+1 ) to estimate the optimal q-function q * (s, a). Here, q * (s, a) is the expected return of selecting action a in state s and following an optimal policy π * . Deep RL methods like DQN  [23]  and DDQN  [35]  represent the q-function as qθ (s, a), where qθ is a neural network whose inputs are features of the state and action, and whose weights θ are updated using stochastic gradient descent.
 Q-learning In partially observable problems, the underlying environment model is typically assumed to be a Partially Observable Markov Decision Process (POMDP). A POMDP is a tuple P O = S, O, A, r, p, ω, γ , where S, A, r, p, and γ are defined as in an MDP, O is a finite set of observations, and ω(s, o) is the observation probability distribution. At every time step t, the agent is in exactly one state s t ∈ S, executes an action a t ∈ A, receives reward r t = r(s t , a t ), and moves to state s t+1 according to p(s t , a t , s t+1 ). However, the agent does not observe s t+1 , but only receives an observation o t+1 ∈ O. This observation provides the agent a clue about what the state s t+1 ∈ S is via ω. In particular, ω(s t+1 , o t+1 ) is the probability of observing o t+1 from state s t+1  [5] . RL methods cannot be immediately applied to POMDPs because the transition probabilities and reward function are not necessarily Markovian w.r.t. O (though by definition they are w.r.t. S). As such, optimal policies may need to consider the complete history o 0 , a 0 , . . . , a t−1 , o t of observations and actions when selecting the next action. Several partially observable RL methods use a recurrent neural network to compactly represent the history, and then use a policy gradient method to train it. However, when we do have access to a full POMDP model P O , then the history can be summarized into a belief state. A belief state is a probability distribution b t : S → [0, 1] over S, such that b t (s) is the probability that the agent is in state s ∈ S given the history up to time t. The initial belief state is computed using the initial observation o 0 : b 0 (s) ∝ ω(s, o 0 ) for all s ∈ S. The belief state b t+1 is then determined from the previous belief state b t , the executed action a t , and the resulting observation o t+1 as b t+1 (s ) ∝ ω(s , o t+1 ) s∈S p(s, a t , s )b t (s) for all s ∈ S. Since the state transitions and reward function are Markovian w.r.t. b t , the set of all belief states B can be used to construct the belief MDP M B . Optimal policies for M B are also optimal for the POMDP  [5] .
 Reward Machines for Partially Observable Environments In this section, we define RMs for the case of partial observability. We use the following problem as a running example to help explain various concepts. Example 3.1 (The cookie domain). The cookie domain, shown in Figure  1a , has three rooms connected by a hallway. The agent (purple triangle) can move in the four cardinal directions. There is a button in the yellow room that, when pressed, causes a cookie to randomly appear in the red or blue room. The agent receives a reward of +1 for reaching (and thus eating) the cookie and may then go and press the button again. Pressing the button before reaching a cookie will move it to a random location. There is no cookie at the beginning of the episode. This is a partially observable environment since the agent can only see what it is in the room that it currently occupies.  RMs are finite state machines that are used to encode a reward function  [33] . In the case of partial observability, they are defined over a set of propositional symbols P that correspond to a set of high-level features that the agent can detect using a labelling function L : O ∅ × A ∅ × O → 2 P where (for any set X) X ∅ X ∪ {∅}. L assigns truth values to symbols in P given an environment experience e = (o, a, o ) where o is the observation seen after executing action a when observing o. We use L(∅, ∅, o) to assign truth values to the initial observation. We call a truth value assignment of P an abstract observation because it provides a high-level view of the low-level environment observations via the labelling function L. A formal definition of an RM follows: Definition 3.1 (reward machine). Given a set of propositional symbols P, a Reward Machine is a tuple R P = U, u 0 , δ u , δ r where U is a finite set of states, u 0 ∈ U is an initial state, δ u is the state-transition function, δ u : U ×2 P → U , and δ r is the reward-transition function, δ r : U ×2 P → R. RMs decompose problems into a set of high-level states U and define transitions using if-like conditions defined by δ u . These conditions are over a set of binary properties P that the agent can detect using L. For example, in the cookie domain, P = { , , , , , , }. These properties are true (i.e., part of an experience label according to L) in the following situations: , , , or is true if the agent ends the experience in a room of that color; is true if the agent ends the experience in the same room as a cookie; is true if the agent pushed the button with its last action; and is true if the agent ate a cookie with its last action (by moving onto the space where the cookie was). Figure  2  shows three possible RMs for the cookie domain. They all define the same reward signal (1 for eating a cookie and 0 otherwise) but differ in their states and transitions. As a result, they differ with respect to the amount of information about the current domain state that can be inferred from the current RM state, as we will see below. Each RM starts in the initial state u 0 . Edge labels in the figures provide a visual representation of the functions δ u and δ r . For example, label , 1 between state u 2 and u 0 in Figure  2b  represents δ u (u 2 , { , }) = u 0 and δ r (u 2 , { , }) = 1. Intuitively, this means that if the RM is in state u 2 and the agent's experience ended in room immediately after eating the cookie , then the agent will receive a reward of 1 and the RM will transition to u 0 . Notice that any properties not listed in the label are false (e.g. must be false to take the transition labelled , 1 ). We also use multiple labels separated by a semicolon (e.g., " , 0 ; , 0 ") to describe different conditions for transitioning between the RM states, each with their own associated reward. The label o/w, r ("o/w" for "otherwise") on an edge from u i to u j means that that transition will be made (and reward r received) if none of the other transitions from u i can be taken. Let us illustrate the behaviour of an RM using the one shown in Figure  2c . The RM will stay in u 0 until the agent presses the button (causing a cookie to appear), whereupon the RM moves to u 1 . From u 1 the RM may move to u 2 or u 3 depending on whether the agent finds a cookie when it enters another room. It is also possible to associate meaning with being in RM states: u 0 means that there is no cookie available, u 1 means that there is a cookie in some room (either blue or red), etc. When learning a policy for a given RM, one simple technique is to learn a policy π(o, u) that considers the current observation o ∈ O and the current RM state u ∈ U . Interestingly, a partially observable problem might be non-Markovian over O, but Markovian over O × U for some RM R P . This is the case for the cookie domain with the RM from Figure  2c , for example. Q-Learning for RMs (QRM) is another way to learn a policy by exploiting a given RM  [33] . QRM learns one q-function qu (i.e., policy) per RM state u ∈ U . Then, given any sample experience, the RM can be used to emulate how much reward would have been received had the RM been in any one of its states. Formally, experience e = (o, a, o ) can be transformed into a valid experience ( o, u , a, o , u , r) used for updating qu for each u ∈ U , where u = δ u (u, L(e)) and r = δ r (u, L(e)). Hence, any off-policy learning method can take advantage of these "synthetically" generated experiences to update all subpolicies simultaneously. When tabular q-learning is used, QRM is guaranteed to converge to an optimal policy on fullyobservable problems  [33] . However, in a partially observable environment, an experience e might be more or less likely depending on the RM state that the agent was in when the experience was collected. For example, experience e might be possible in one RM state u i but not in RM state u j . Thus, updating the policy for u j using e as QRM does, would introduce an unwanted bias to quj . We will discuss how to (partially) address this problem in §5. u 0 , 1 ; , 1 ; o/w, 0 (a) Naive RM. u 0 u 1 u 2 o/w, 0 o/w, 0 o/w, 0 , 0 , 0 ; , 0 , 1 , 1 (b) "Optimal" RM. u 0 u 1 u 2 u 3 o/w, 0 o/w, 0 o/w, 0 o/w, 0 , 0 , 0 ; , 0 , 0 ; , 0 , 1 , 1 , 0 , 0 (c) Perfect RM.
 Learning Reward Machines from Traces Our overall idea is to search for an RM that can be used as external memory by an agent for a given task. As input, our method will only take a set of high-level propositional symbols P, and a labelling function L that can detect them. Then, the key question is what properties should such an RM have. Three proposals naturally emerge from the literature. The first comes from the work on learning Finite State Machines (FSMs)  [3, 40, 10] , which suggests learning the smallest RM that correctly mimics the external reward signal given by the environment, as in Giantamidis and Tripakis' method for learning Moore Machines  [10] . Unfortunately, such approaches would learn RMs of limited utility, like the one in Figure  2a . This naive RM correctly predicts reward in the cookie domain (i.e., +1 for eating a cookie , zero otherwise) but provides no memory in support of solving the task. The second proposal comes from the literature on learning Finite State Controllers (FSC)  [22]  and on model-free RL methods  [32] . This work suggests looking for the RM whose optimal policy receives the most reward. For instance, the RM from Figure  2b  is "optimal" in this sense. It decomposes the problem into three states. The optimal policy for u 0 goes directly to press the button, the optimal policy for u 1 goes to the blue room and eats the cookie if present, and the optimal policy for u 2 goes to the red room and eats the cookie. Together, these three policies give rise to an optimal policy for the complete problem. This is a desirable property for RMs, but requires computing optimal policies in order to compare the relative quality of RMs, which seems prohibitively expensive. However, we believe that finding ways to efficiently learn "optimal" RMs is a promising future work direction. Finally, the third proposal comes from the literature on Predictive State Representations (PSR)  [20] , Deterministic Markov Models (DMMs)  [21] , and model-based RL  [16] . These works suggest learning the RM that remembers sufficient information about the history to make accurate Markovian predictions about the next observation. For instance, the cookie domain RM shown in Figure  2c  is perfect w.r.t. this criterion. Intuitively, every transition in the cookie environment is already Markovian except for transitioning from one room to another. Depending on different factors, when entering to the red room there could be a cookie there (or not). The perfect RM is able to encode such information using 4 states: when at u 0 the agent knows that there is no cookie, at u 1 the agent knows that there is a cookie in the blue or the red room, at u 2 the agent knows that there is a cookie in the red room, and at u 3 the agent knows that there is a cookie in the blue room. Since keeping track of more information will not result in better predictions, this RM is perfect. Below, we develop a theory about perfect RMs and describe an approach to learn them.
 Perfect Reward Machines: Formal Definition and Properties The key insight behind perfect RMs is to use their states U and transitions δ u to keep track of relevant past information such that the partially observable environment P O becomes Markovian w.r.t. O × U . Definition 4.1 (perfect reward machine). An RM R P = U, u 0 , δ u , δ r is considered perfect for a POMDP P O = S, O, A, r, p, ω, γ with respect to a labelling function L if and only if for every trace o 0 , a 0 , . . . , o t , a t generated by any policy over P O , the following holds: Pr(o t+1 , r t |o 0 , a 0 , . . . , o t , a t ) = Pr(o t+1 , r t |o t , x t , a t ) (1) where x 0 = u 0 and x t = δ u (x t−1 , L(o t−1 , a t−1 , o t )) . Two interesting properties follow from Definition 4.1. First, if the set of belief states B for the POMDP P O is finite, then there exists a perfect RM for P O with respect to some L. Second, the optimal policies for perfect RMs are also optimal for the POMDP (see supplementary material §12). Instead, we propose an alternative that focuses on a necessary condition for a perfect RM: the RM must predict what is possible and impossible in the environment at the abstract level. For example, it is impossible to be at u 3 in the RM from Figure  2c  and make the abstract observation { , }, because the RM reaches u 3 only if the cookie was seen in the blue room or not to be in the red room. This idea is formalized in the optimization model LRM. Let T = {T 0 , . . . , T n } be a set of traces, where each trace T i is a sequence of observations, actions, and rewards: T i = (o i,0 , a i,0 , r i,0 , . . . , a i,ti−1 , r i,ti−1 , o i,ti ). (2) We now look for an RM U, u 0 , δ u , δ r that can be used to predict L(e  |U | ≤ umax (4) xi,t ∈ U ∀i ∈ I, t ∈ Ti ∪ {ti} (5) xi,0 = u0 ∀i ∈ I (6) xi,t+1 = δu(xi,t, L(ei,t+1)) ∀i ∈ I, t ∈ Ti (7) N u,l ⊆ 2 2 P ∀u ∈ U, l ∈ 2 P (8) L(ei,t+1) ∈ N x i,t ,L(e i,t ) ∀i ∈ I, t ∈ Ti (9) Constraints (  3 ) and (  4 ) ensure that we find a well-formed RM over P with at most u max states. Constraint (  5 ),  (6) , and (7) ensure that x i,t is equal to the current state of the RM, starting from u 0 and following δ u . Constraint (  8 ) and (  9 ) ensure that the sets N u,l contain every L(e i,t+1 ) that has been seen right after l and u in T . The objective function comes from maximizing the log-likelihood for predicting L(e i,t+1 ) using a uniform distribution over all the possible options given by N u,l . A key property of this formulation is that any perfect RM is optimal w.r.t. the objective function in LRM when the number of traces tends to infinity (see supplementary material §12): Theorem 4.3. When the set of training traces (and their lengths) tends to infinity and is collected by a policy such that π(a|o) > for all o ∈ O and a ∈ A, any perfect RM with respect to L and at most u max states will be an optimal solution to the formulation LRM. Finally, note that the definition of a perfect RM does not impose conditions over the rewards associated with the RM (i.e., δ r ). This is why δ r is a free variable in the model LRM. However, we still expect δ r to model the external reward signals given by the environment. To do so, we estimate δ r (u, l) using its empirical expectation over T (as commonly done when constructing belief MDPs  [5] ).
 Searching for a Perfect Reward Machine Using Tabu Search We now describe the specific optimization technique used to solve LRM. We experimented with many discrete optimization approaches-including mixed integer programming  [6] , Benders decomposition  [8] , evolutionary algorithms  [17] , among others-and found local search algorithms  [1]  to be the most effective at finding high quality RMs given short time limits. In particular, we use Tabu search  [11] , a simple and versatile local search procedure with convergence guarantees and many successful applications in the literature  [36] . We also include our unsuccessful mixed integer linear programming model for LRM in the supplementary material §10. In the context of our work, Tabu search starts from a random RM and, on each iteration it evaluates all "neighbouring" RMs. We define the neighbourhood of an RM as the set of RMs that differ by exactly one transition (i.e., removing/adding a transition, or changing its value) and evaluate RMs using the objective function of LRM. When all neighbouring RMs are evaluated, the algorithm chooses the one with lowest values and sets it as the current RM. To avoid local minima, Tabu search maintains a Tabu list of all the RMs that were previously used as the current RM. Then, RMs in the Tabu list are pruned when examining the neighbourhood of the current RM.
 Simultaneously Learning a Reward Machine and a Policy We now describe our overall approach to simultaneously finding an RM and exploiting that RM to learn a policy. The complete pseudo-code can be found in the supplementary material (Algorithm 1). Our approach starts by collecting a training set of traces T generated by a random policy during t w "warmup" steps. This set of traces is used to find an initial RM R using Tabu search. The algorithm then initializes policy π, sets the RM state to the initial state u 0 , and sets the current label l to the initial abstract observation L(∅, ∅, o). The standard RL learning loop is then followed: an action a is selected following π(o, u) where u is the current RM state, and the agent receives the next observation o and the immediate reward r. The RM state is then updated to u = δ u (u, L(o, a, o )) and the last experience ( o, u , a, r, o , u ) is used by any RL method of choice to update π. Note that in an episodic task, the environment and RM are reset whenever a terminal state is reached. If on any step, there is evidence that the current RM might not be the best one, our approach will attempt to find a new one. Recall that the RM R was selected using the cardinality of its prediction sets N (LRM). Hence, if the current abstract observation l is not in N u,l , adding the current trace to T will increase the size of N u,l for R. As such, the cost of R will increase and it may no longer be the best RM. Thus, if l ∈ N u,l , we add the current trace to T and search for a new RM. Recall that we use Tabu search, though any discrete optimization method could be applied. Our method only uses the new RM if its cost is lower than R's. If the RM is updated, a new policy is learned from scratch. Given the current RM, we can use any RL algorithm to learn a policy π(o, u), by treating the combination of o and u as the current state. If the RM is perfect, then the optimal policy π * (o, u) will also be optimal for the original POMDP (as stated in Theorem 4.2). However, to exploit the problem structure exposed by the RM, we can use the QRM algorithm. As explained in §3, standard QRM under partial observability can introduce a bias because an experience e = (o, a, o ) might be more or less likely depending on the RM state that the agent was in when the experience was collected. We partially address this issue by updating qu using (o, a, o ) if and only if L(o, a, o ) ∈ N u,l , where l was the current abstract observation that generated the experience (o, a, o ). Hence, we do not transfer experiences from u i to u j if the current RM does not believe that (o, a, o ) is possible in u j . For example, consider the cookie domain and the perfect RM from Figure  2c . If some experience consists of entering to the red room and seeing a cookie, then this experience will not be used by states u 0 and u 3 as it is impossible to observe a cookie at the red room from those states. Note that adding this rule may work in many cases, but it will not address the problem in all environments (more discussion in §7). We consider addressing this problem as an interesting area for future work.
 Experimental Evaluation We tested our approach on three partially observable grid domains (Figure  1 ). The agent can move in the four cardinal directions and can only see what is in the current room. These are stochastic domains where the outcome of an action randomly changes with a 5% probability. The first environment is the cookie domain (Figure  1a ) described in §3. Each episode is 5, 000 steps long, during which the agent should attempt to get as many cookies as possible. The second environment is the symbol domain (Figure  1b ). It has three symbols ♣, ♠, and in the red and blue rooms. One symbol from {♣, ♠, } and possibly a right or left arrow are randomly placed at the yellow room. Intuitively, that symbol and arrow tell the agent where to go, e.g., ♣ and → tell the agent to go to ♣ in the east room. If there is no arrow, the agent can go to the target symbol in either room. An episode ends when the agent reaches any symbol in the red or blue room, at which point it receives a reward of +1 if it reached the correct symbol and −1 otherwise. The third environment is the 2-keys domain (Figure  1c ). The agent receives a reward of +1 when it reaches the coffee (in the yellow room). To do so, it must open the two doors (shown in brown). Each door requires a different key to open it, and the agent can only carry one key at a time. Initially, the two keys are randomly located in either the blue room, the red room, or split between them. We tested two versions of our Learned Reward Machine (LRM) approach: LRM+DDQN and LRM+DQRM. Both learn an RM from experience as described in §4.2, but LRM+DDQN learns a policy using DDQN  [35]  while LRM+DQRM uses the modified version of QRM described in §5. In all domains, we used u max = 10, t w = 200, 000, an epsilon greedy policy with = 0.1, and a discount factor γ = 0.9. The size of the Tabu list and the number of steps that the Tabu search performs before returning the best RM found is 100. We compared against 4 baselines: DDQN  [35] , A3C  [24] , ACER  [37] , and PPO  [29]  using the OpenAI baseline implementations  [12] . DDQN uses the concatenation of the last 10 observations as input which gives DDQN a limited memory to better handle the domains. A3C, ACER, and PPO use an LSTM to summarize the history. Note that the output of the labelling function was also given to the baselines. Details on the hyperparameters and networks can be found in the supplementary material §13. Figure  3  shows the total cumulative rewards that each approach gets every 10, 000 training steps and compares it to the optimal policy. For the LRM algorithms, the figure shows the median performance over 30 runs per domain, and percentile 25 to 75 in the shadowed area. For the DDQN baseline, we show the maximum performance seen for each time period over 5 runs per problem. Similarly, we also show the maximum performance over the 30 runs of A3C, ACER, and PPO per period. All the baselines outperformed a random policy, but none make much progress on any of the domains. Furthermore, LRM approaches largely outperform all the baselines, reaching close-to-optimal policies in the cookie and symbol domain. We also note that LRM+DQRM learns faster than LRM+DDQN, but is more unstable. In particular, LRM+DQRM converged to a considerably better policy than LRM+DDQN in the 2-keys domain. We believe this is due to QRM's experience sharing mechanism that allows for propagating sparse reward backwards faster (see supplementary material §13.3). A key factor in the strong performance of the LRM approaches is that Tabu search finds high-quality RMs in less than 100 local search steps (Figure  5 , supplementary material). In fact, our results show that Tabu search finds perfect RMs in most runs, in particular when tested over the symbol domain. 7 Discussion, Limitations, and Broader Potential Solving partially observable RL problems is challenging and LRM was able to solve three problems that were conceptually simple but presented a major challenge to A3C, ACER, and PPO with LSTM-based memories. A key idea behind these results was to optimize over a necessary condition for perfect RMs. This objective favors RMs that are able to predict possible and impossible future observations at the abstract level given by the labelling function L. In this section, we discuss the advantages and current limitations of such an approach. u 0 u 1 , 1 ; o/w, 0 , 1 ; o/w, 0 , 0 , 0 We begin by considering the performance of Tabu search in our domains. Given a training set composed of one million transitions, a simple Python implementation of Tabu search takes less than 2.5 minutes to learn an RM across all our environments, when using 62 workers on a Threadripper 2990WX processor. Note that Tabu search's main bottleneck is evaluating the neighbourhood around the current RM solution. As the size of the neighbourhood depends on the size of the set of propositional symbols P, exhaustively evaluating the neighbourhood may sometimes become impractical. To handle such problem, it will be necessary to import ideas from the Large Neighborhood Search literature  [27] . Regarding limitations, learning the RM at the abstract level is efficient but requires ignoring (possibly relevant) low-level information. For instance, Figure  4  shows an adversarial example for LRM. The agent receives reward for eating the cookie ( ). There is an external force pulling the agent down-i.e., the outcome of the "move-up" action is actually a downward movement with high probability. There is a button ( ) that the agent can press to turn off (or back on) the external force. Hence, the optimal policy is to press the button and then eat the cookie. Given P = { , }, a perfect RM for this environment is fairly simple (see Figure  4 ) but LRM might not find it. The reason is that pressing the button changes the low-level probabilities in the environment but does not change what is possible or impossible at the abstract level. In other words, while the LRM objective optimizes over necessary conditions for finding a perfect RM, those conditions are not sufficient to ensure that an optimal solution will be a perfect RM. In addition, if a perfect RM is found, our heuristic approach to share experiences in QRM would not work as intended because the experiences collected when the force is on (at u 0 ) would be used to update the policy for the case where the force is off (at u 1 ). Other current limitations include that it is unclear how to handle noise over the high-level detectors L and how to transfer learning from previously learned policies when a new RM is learned. Finally, defining a set of proper high-level detectors for a given environment might be a challenge to deploying LRM. Hence, looking for ways to automate that step is an important direction for future work.
 Related Work State-of-the-art approaches to partially observable RL use Recurrent Neural Networks (RNNs) as memory in combination with policy gradient  [24, 37, 29, 15] , or use external neural-based memories  [25, 18, 13] . Other approaches include extensions to Model-Based Bayesian RL to work under partial observability  [28, 7, 9]  and to provide a small binary memory to the agent and a special set of actions to modify it  [26] . While our experiments highlight the merits of our approach w.r.t. RNN-based approaches, we rely on ideas that are largely orthogonal. As such, we believe there is significant potential in mixing these approaches to get the benefit of memory at both the high-and the low-level. The effectiveness of automata-based memory has long been recognized in the POMDP literature  [5] , where the objective is to find policies given a complete specification of the environment. The idea is to encode policies using Finite State Controllers (FSCs) which are FSMs where the transitions are defined in terms of low-level observations from the environment and each state in the FSM is associated with one primitive action. When interacting with the environment, the agent always selects the action associated with the current state in the controller. Meuleau et al.  [22]  adapted this idea to work in the RL setting by exploiting policy gradient to learn policies encoded as FSCs. RMs can be considered as a generalization of FSC as they allow for transitions using conditions over high-level events and associate complete policies (instead of just one primitive action) to each state. This allows our approach to easily leverage existing deep RL methods to learn policies from low-level inputs, such as images-which is not achievable by Meuleau et al.  [22] . That said, further investigating using ideas for learning FSMs  [3, 40, 10]  in learning RMs is a promising direction for future work. Our approach to learn RMs is greatly influenced by Predictive State Representations (PSRs)  [20] . The idea behind PSRs is to find a set of core tests (i.e., sequences of actions and observations) such that if the agent can predict the probabilities of these occurring, given any history H, then those probabilities can be used to compute the probability of any other test given H. The insight is that state representations that are good for predicting the next observation are good for solving partially observable environments. We adapted this idea to the context of RM learning as discussed in §4. While our work was under review, two interesting papers were submitted to arXiv. The first paper, by Xu et al.  [39] , proposes a polynomial time algorithm to learn reward machines in fully observable domains. Their goal is to learn the smallest reward machine that is consistent with the reward function-which makes sense for fully observable domains, but would have limited utility under partial observability (as discussed in §4). The second paper, by Zhang et al.  [41] , proposes to learn a discrete PSR representation of the environment directly from low-level observations and then plan over such representation using tabular Q-learning. This is a promising research direction, with some clear synergies with LRM.
 Concluding Remarks We have presented a method for learning (perfect) Reward Machines in partially observable environments and demonstrated the effectiveness of these learned RMs in tackling partially observable RL problems that are unsolvable by A3C, ACER and PPO. Informed by criteria from the POMDP, FSC, and PSR literature, we proposed a set of RM properties that support tackling RL in partially observable environments. We used these properties to formulate RM learning as a discrete optimization problem. We experimented with several optimization methods, finding Tabu search to be the most effective. We then combined this RM learning with policy learning for partially observable RL problems. Our combined approach outperformed a set of strong LSTM-based approaches on different domains. We believe this work represents an important building block for creating RL agents that can solve cognitively challenging partially observable tasks. Not only did our approach solve problems that were unsolvable by A3C, ACER and PPO, but it did so in a relatively small number of training steps. RM learning provided the agent with memory, but more importantly the combination of RM learning and policy learning provided it with discrete reasoning capabilities that operated at a higher level of abstraction, while leveraging deep RL's ability to learn policies from low-level inputs. This work leaves open many interesting questions relating to abstraction, observability, and properties of the language over which RMs are constructed. We believe that addressing these questions, among many others, will push the boundary of partially observable RL problems that can be solved.
 Algorithm for Simultaneously Learning Reward Machines and a Policy Algorithm 1 shows our overall approach to simultaneously learning an RM and exploiting that RM to learn a policy. The algorithm inputs are the set of propositional symbols P, the labelling function L, a maximum on the number of RM states u max , and the number of "warmup" steps t w . Our approach starts by collecting a training set of traces T generated by a random policy during t w steps (Line 2). This set of traces is used to find an initial RM R using Tabu search (Line 3). If later traces show that R is incorrect, our method will then find a new RM learned using the additional traces. Lines 4 and 5 initialize the environment and the policy π, and set variables x and l to the initial RM state u 0 and initial abstract observation L(∅, ∅, o), respectively. Lines 7-19 are the main loop of our approach. Lines 7-10 are part of the standard RL loop: the agent executes an action a selected following π(o, x) and receives the next observation o , the immediate reward r, and a boolean variable done indicating if the episode has terminated. Then, the state in the RM x is updated and the policy π is improved using the last experience ( o, x , a, r, o , x , done). Note that when done is true, the environment and RM are reset (Lines 17-18). Lines 11-16 involve relearning the RM when there is evidence that the current RM might not be the best one. Recall that the RM R was selected using the cardinality of its prediction sets N (see the description of LRM). Hence, if the current abstract observation l is not in N x,l , then adding the current trace to T will increase the size of N x,l for R. As such, the cost of R will increase and it may no longer be the best RM. Thus, if l ∈ N x,l , we add the current trace to T and use Tabu search to find a new RM. Note, our method only uses the new RM if its cost is lower than that of R (Lines 14-16). However, when the RM is updated, a new policy is learned from scratch (Line 16). T ← T ∪ get_current_trace() Proof sketch. If the reachable belief space B is finite, we can construct an RM that keeps track of the current belief state using one RM state per belief state and emulating their progression using δ u , and one propositional symbol for every action-observation pair. Thus, the current belief state b t can be inferred from the last observation, last action, and the current RM state. As such, the equality from Definition 4.1 holds. Two interesting properties follow from the definition of a perfect RM. First, if the set of belief states B for the POMDP P O is finite, then there exists a perfect RM for P O with respect to some L. Second, the optimal policies for perfect RMs are also optimal for the POMDP. Theorem 12.2. Let R P be a perfect RM for a POMDP P O w.r.t. a labelling function L, then any optimal policy for R P w.r.t. the environmental reward is also optimal for P O . Proof sketch. As the next observation and immediate reward probabilities can be predicted from O × U × A, an optimal policy over O × U must also be optimal over P O . A key property of this formulation is that any perfect RM is optimal with respect to the objective function in LRM when the number of traces tends to infinity: Theorem 12.3. When the set of training traces (and their lengths) tends to infinity and is collected by a policy such that π(a|o) > for all o ∈ O and a ∈ A, and some constant > 0, then any perfect RM with respect to L and at most u max states will be an optimal solution to the formulation given in LRM. Proof sketch. In the limit, l ∈ N u,l if and only if the probability of observing l after executing an action from the RM state u while observing l is non-zero. In particular, for all i ∈ I and t ∈ T , the cardinality of N xi,t,L(ei,t) will be minimal for a perfect RM. This follows from the fact that perfect RMs make perfect predictions for the next observation o given o, u, and a. Therefore, as we minimize the sum over log(|N xi,t,L(ei,t) |), the objective value for a perfect RM must be minimal.
 Experimental Evaluation 13.1 Experimental Details For LRM+DDQN and LRM+DQRM, the neural network used has 5 fully connected layers with 64 neurons per layer. On every step, we trained the network using 32 sampled experiences from a replay buffer of size 100,000 using the Adam optimizer  [19]  and a learning rate of 5e-5. The target networks were updated every 100 steps. DDQN  [35]  uses the same parameters and network architecture as LRM+DDQN, but its input is the concatenation of the last 10 observations, as commonly done by Atari playing agents. This gives DDQN a limited memory to better handle partially observable domains. In contrast, A3C, ACER, and PPO use an LSTM to summarize the history. We also followed the same testing methodology that was used in their original publications. We ran each approach at least 30 times per domain, and on every run, we randomly selected the number of hidden neurons for the LSTM from {64, 128, 256, 512} and a learning rate from (1e-3, 1e-5). We also sampled δ from {0, 1, 2} for ACER and the clip range from (0.1, 0.3) for PPO. Other parameters were fixed to their default values. While interacting with the environment, the agents were given a "top-down" view of the world represented as a set of binary matrices. One matrix had a 1 in the current location of the agent, one had a 1 in only those locations that are currently observable, and the remaining matrices each corresponded to an object in the environment and had a 1 at only those locations that were both currently observable and contained that object (i.e., locations in other rooms are "blacked out"). The agent also had access to features indicating if they were carrying a key, which colour room they were in, and the current status (i.e., occurring or not occurring) of the events detected by the labelling function.
 Tabu Search Figure  5  evaluates the quality of the RMs found by Tabu search by comparing it the perfect RM. In each plot, a dot compares the cost of a handcrafted perfect RM with that of an RM R that was found by Tabu search while running our LRM approaches, where the costs are evaluated relative to the training set used to find R. Being on or under the diagonal line (as in most of the points in the figure) means that Tabu search is finding RMs whose values are at least as good as the handcrafted RM. Hence, Tabu search is either finding perfect RMs or discovering that our training set is incomplete and our agent will eventually fill those gaps.
 Cookie domain Symbol domain 2-keys domains Cost Perfect RM Cost Learned RM Cost Perfect RM Cost Learned RM Cost Perfect RM Cost Learned RM
 DDQN vs DQRM: Exploration Heatmaps and Learned Trajectories As shown in §6 of the paper, LRM+DQRM tends to learn faster than LRM+DDQN and largely outperforms LRM+DDQN in the 2-keys domain. Towards better understating these results, we ran the following experiment. For the 2-keys domain (and identical random seed), we learn policies using DDQN and DQRM over the same handcrafted perfect RM from Figure  6 . RM state u 6 is fairly simple: when the agent is carrying a key and there is only one closed door, go and get the coffee. As expected, DQRM seems to learn an accurate estimation of that q-function using very limited interactions within the coffee room. In contrast, DDQN uses one big q-network to model the complete policy. Receiving reward by getting the coffee pushes the q-network estimation to believe that a high reward can be collected from the coffee room (even if the doors are closed). Hence, the agent spent a considerable amount of time hitting the second door without having a key. Second, DQRM shares experience over all the q-networks. This allows for using the experiences collected while being at early stages of the task (e.g., states u 0 , u 1 , u 2 , and u 3 ) to update policies for later stages (e.g., states u 4 , u 5 , and u 6 ). In particular, all the experience needed to learn to navigate from one room to another is shared. Therefore, while DDQN would depend on its network to avoid relearning how to navigate between rooms when the RM state changes, DQRM enforces such transfer by sharing experiences whenever it is appropriate. This might explain why DDQN spends considerably more time in the hallway than DQRM in heatmaps 50-100K and 100-150K.  Note that the exploratory trend from 150-500K follows a clear pattern. On one hand, the DQRM agent seems to have a good idea of how to solve the task and, therefore, it spends most of its time on the hallway (solving the tasks requires passing through the hallway at least 4 times). On the other hand, the DDQN agent is getting stuck exploring subregions of the map. Finally, we inspected the trajectories of each agent when solving the task after 1 million training steps. Figures 9 shows the DDQN agent and Figure  10  shows the DQRM agent. Both agents solved the task, but DQRM solved it faster (83 steps vs 102 steps). In solving this problem, the main difficulty for the DDQN agent is in reacting to entering the South room and discovering that the keys are not there. Its reaction is to enter and leave the empty room many times before going to the North room. In the case of DQRM, the agent goes directly to the North room after observing that the South room is empty, but then it enters and leaves the North room a few times before collecting the keys. After collecting the first key, both agents solved the rest of the task almost optimally.    Figure 1 : 1 Figure 1: Partially observable environments where the agent can only see what is in the current room.
 Figure 2 : 2 Figure 2: Three possible Reward Machines for the Cookie domain.
 Figure 3 : 3 Figure 3: Total reward collected every 10, 000 training steps.
 Figure 4 : 4 Figure 4: The gravity domain.
 Algorithm 1 1 Learning an RM and a Policy 1: Input: P, L, A, u max , t w 2: T ← collect_traces(t w ) 3: R, N ← learn_rm(P, L, T , u max ) 4: o, x, l ← env_get_initial_state(), u 0 , L(∅, ∅, o) 5: π ← initialize_policy() 6: for t = 1 to t train do7: a ← select_action(π, o, x) 8: o , r, done ← env_execute_action(a) 9: x , l ← δ u (x, L(o, a, o )), L(o, a, o ) 10: π ← improve(π, o, x, l, a, r, o , x , l , done, N ) 11: if l ∈ N x,l then 12:
 Figure 5 : 5 Figure 5: Cost comparison between perfect RM and RM found by Tabu search.
 steps 300-350K steps 350-400K steps 400-450K steps 450-500K steps DQRM DQRM DQRM DQRM DQRM 250-300K steps 300-350K steps 350-400K steps 400-450K steps 450-500K steps
 Figure 8 : 8 Figure 8: Exploration heatmaps for DDQN and DQRM over the 2-keys domain given a perfect RM.
 (a) Collecting the first key. (b) Opening the first door. (c) Collecting the other key. (d) Opening the second door.
 Figure 9 : 9 Figure 9: The learned trace after 1 million training steps by DDQN given a perfect RM, divided into four stages.
 Figure 10 : 10 Figure 10: The learned trace after 1 million training steps by QRM given a perfect RM, divided into four stages.
 i,t+1 ) from L(e i,t ) and the current RM state x i,t , where e i,t+1 is the experience (o i,t , a i,t , o i,t+1 ) and e i,0 is (∅, ∅, o i,0 ) by definition. The model parameters are the set of traces T , the set of propositional symbols P, the labelling function L, and a maximum number of states in the RM u max . The model also uses the sets I = {0 . . . n} and T i = {0 . . . t i − 1}, where I contains the index of the traces and T i their time steps. The model has two auxiliary variables x i,t and N u,l . Variable x i,t ∈ U represents the state of the RM after observing trace T i up to time t. Variable N u,l ⊆ 2 2 P is the set of all the next abstract observations seen from the RM state u and the abstract observations l at some point in T . In other words, l ∈ N u,l iff u = x i,t , l = L(e i,t ), and l = L(e i,t+1 ) for some trace T i and time t. minimize U,u 0 ,δu,δr i∈I t∈T i log(|N x i,t ,L(e i,t ) |) (LRM) s.t. U, u0, δu, δr ∈ RP (3)
 Given any POMDP P O with a finite reachable belief space, there exist a perfect RM for P O with respect to some labelling function L. 12 Theorems and Proof Sketches Theorem 12.1. 13: 14: 15: 16: 17: R , N ← relearn_rm(R, P, L, T , u max ) if R = R then R, done ← R , true π ← initialize_policy() end if 18: 19: 20: 21: end if if done then o , x , l ← env_get_initial_state(), u 0 , L(∅, ∅, o) end if 22: 23: end for o, x, l ← o , x , l 24: return π

Abstract: Reward Machines (RMs) provide a structured, automata-based representation of a reward function that enables a Reinforcement Learning (RL) agent to decompose an RL problem into structured subproblems that can be efficiently learned via off-policy learning. Here we show that RMs can be learned from experience, instead of being specified by the user, and that the resulting problem decomposition can be used to effectively solve partially observable RL problems. We pose the task of learning RMs as a discrete optimization problem where the objective is to find an RM that decomposes the problem into a set of subproblems such that the combination of their optimal memoryless policies is an optimal policy for the original problem. We show the effectiveness of this approach on three partially observable domains, where it significantly outperforms A3C, PPO, and ACER, and discuss its advantages, limitations, and broader potential.  1