2020-21 New Year review

This is an annual post reviewing the last year and making resolutions and predictions for next year. 2020 brought a combination of challenges from living in a pandemic and becoming a parent. Other highlights include not getting sick, getting a broader perspective on my life through decluttering, and going back to Ukraine for the first time. (This post was written in bits and pieces over the past two months.)

#### 2020 review

*Life updates:*

Janos and I had a son, Daniel, on Nov 11. He arrived almost 3 weeks later than expected (apparently he was waiting to be born on my late grandfather’s birthday), and has been a great source of cuddles, sound effects and fragmented sleep ever since.

[![](https://vkrakovna.files.wordpress.com/2021/01/20201117_113230.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/20201117_113230.jpg)1 week old
[![](https://vkrakovna.files.wordpress.com/2021/01/20201224_113030.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/20201224_113030.jpg)6 weeks old
Some work things also went well this year – I had a paper accepted at NeurIPS, and was promoted to senior research scientist. Also, I did not get covid, and survived half a year of working from home (much credit goes to the great company of my housemates). Overall, a lot of things to be grateful for.

*AI safety research:*

* Wrote a paper on [avoiding side effects by considering future tasks](https://arxiv.org/abs/2010.07877), providing some theoretical grounding for the side effects problem, which was accepted to NeurIPS 2020.
* Contributed theoretical results for a project on the tampering problem and coauthored two papers: [Avoiding Tampering Incentives in Deep RL via Decoupled Approval](https://arxiv.org/abs/2011.08827) and [REALab: An Embedded Perspective on Tampering](https://arxiv.org/abs/2011.08820).
* Wrote a blog post on the specification gaming problem for the DeepMind blog: [Specification gaming: the flip side of AI ingenuity](https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity).

*Effectiveness:*

Working from home was definitely a productivity hit. I was mostly focused on urgent tasks, such as conference submissions and reviewing, and didn’t get much research done. I did 366 hours of “deep work” (1.95 hours per work day) this year, compared to 551 hours (2.4 hours per work day) in 2019. This includes theory work, reading papers, writing papers and code, but not editing text or debugging. I got back into using work cycles, which added some helpful structure in the home environment.

[![](https://vkrakovna.files.wordpress.com/2022/12/image-4.png?w=546)](https://vkrakovna.files.wordpress.com/2022/12/image-4.png)
I was very grateful to be living in a group house during the pandemic. While it was a bit tricky to have 5 people sharing the space when working from home, it was awesome to have an in-person community and not feel completely isolated from the world. It was also much easier to do nonzero exercise when I had someone to do it with, e.g. running in the park together.

Spending a lot of time at home inspired me to do a lot of decluttering. In particular, I went through all my old notes, got rid of most of them, and gathered the ones that still seem interesting and relevant (notes from rationality workshops, Hamming worksheets, reflections and so on). I put these into a binder for easy reading, and found it useful for getting a big picture sense of how my attitudes and problems have evolved over time. This has been particularly helpful during the pandemic, when my life has often felt small and repetitive.

I got a UK driving license for automatic cars, which took a surprising amount of practice given that I already had a US license. There was a lot to get used to with the left side of the road, the narrowness of the streets and frequent maneuvering – I spent a number of lessons just on getting the positioning right. I did the theory test in March and planned to take the road test in early summer, but then the driving schools closed for lockdown, and I ended up starting the lessons in July. I took the road test in September and didn’t pass because of “undue hesitation” at a busy roundabout, so I had to repeat the test in October, two weeks before Daniel was due (thankfully, I passed this time and could forget about driving for a while).

*Health:*

Physical health has been pretty good this year. Last year I had 7 colds, while this year I was not sick at all – probably due to social distancing and taking zinc regularly. Thankfully, recovery after the birth was relatively quick, feeling mostly normal in around 2 weeks, though it will take some time to get my core muscles back online. I’ve been getting some back pain from lifting Daniel (now at 6kg thanks to his voracious appetite), which makes it all the more important to rebuild core strength.

The second half of the year came with pretty bad sleep – a lot of insomnia in the last trimester where I woke up at 3-4am for no discernible reason and couldn’t fall asleep again, followed by fragmented sleep after Daniel was born. Living and working on 4-5 hours of sleep before I went on parental leave was surprisingly ok, probably because I was waking up on my own rather than being woken up in the middle of a sleep cycle. On the other hand, being woken up by a hungry baby definitely feels more meaningful than waking up at 3am for no reason and not being able to go back to sleep.

[![](https://vkrakovna.files.wordpress.com/2021/01/insomnia.png?w=380)](https://vkrakovna.files.wordpress.com/2021/01/insomnia.png)Rate of insomnia by month

[![](https://vkrakovna.files.wordpress.com/2021/01/hours-awake.png?w=380)](https://vkrakovna.files.wordpress.com/2021/01/hours-awake.png)Average hours awake at night by month

[![](https://vkrakovna.files.wordpress.com/2021/01/hours-sleep.png?w=365)](https://vkrakovna.files.wordpress.com/2021/01/hours-sleep.png)Average hours of sleep by month
This year has been pretty hard on my mental health due to a number of ways that the pandemic interacted with having a kid, and various problems that I considered solved have made a comeback lately. I spent most of the year at home without many forms of self-care, such as my usual exercise, sufficient sleep, or nice things like going to the sauna. While the birth went well, there were a lot more stressful interactions with the healthcare system than I had hoped. After that, there has been a combination of sleep deprivation, limited daylight, a mostly empty house after some housemates moved to the countryside, difficulty with meeting friends outside because of cold weather, making increasingly modest plans only to have them shot down by the ever-changing lockdown rules, and the exciting new covid strain we have in London that calls for high levels of caution and isolation. Thankfully, my mom was able to come stay with us for a few months to help with the baby, make food and keep us company.

*Travel:*

In January I visited Ukraine for the first time since I emigrated 17 years ago. I saw my dad and aunt, as well as my niece and her kids who live in a remote part of Canada but happened to be in Ukraine for the winter. It was an interesting experience to navigate around Kyiv – I no longer had a map of the city in my head, so I recognized some familiar places but could not recall where they are relative to each other, so this felt like visiting a new city with a lot of deja vu. I was pleasantly surprised by the large number of Georgian restaurants in Kyiv, which we made sure to frequent and were not disappointed.

[![](https://vkrakovna.files.wordpress.com/2021/01/img_20200113_142052.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/img_20200113_142052.jpg)
In March we did a week-long meditation retreat at MAPLE in the US. We hesitated whether to go ahead with this plan given that flights might get canceled, but ultimately decided to go. The retreat was in a remote location in Vermont that seemed pretty safe from a covid perspective. I was advised to follow an equanimity practice that worked pretty well (focusing on acceptance rather than observation of things that come into my awareness).

[![](https://vkrakovna.files.wordpress.com/2021/01/20200312_092355.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/20200312_092355.jpg)Janos in his element
We had a peaceful week meditating among the snows, which unfortunately became less peaceful at the end when some people started coughing, so we spent the last couple of days meditating in masks and gloves, and left as soon as the retreat ended. Our flight back did get canceled, but we were rebooked on another one for free. Upon returning home, we self-isolated from our housemates and acquired covid tests, which were thankfully negative. We later learned that several people at the retreat tested positive for covid, so this was a close call.

In August we went camping in North Wales at a friend’s cottage (after the first lockdown was lifted). The cottage itself was abandoned (and a bit spooky), but we could stay on the adjacent land and thus avoid crowds at the newly opened campsites. We enjoyed a lot of swimming in cold waterfalls and a much warmer Atlantic ocean. We also hiked up a nearby mountain Arenig Fawr (Daniel and I were taking it slow).

[![](https://vkrakovna.files.wordpress.com/2021/01/20200802_142928.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/20200802_142928.jpg)View from the summit of Arenig Fawr
In September we had a vacation in Madeira, where everyone was tested for covid on arrival at the airport, and there was no community transmission at the time. Madeira is a volcanic island that is basically one big mountain, and we had an interesting time driving around it (on the way to a hike, our car refused to go up a very steep road and we took a taxi the rest of the way). The terrain was a great combination of ocean and mountains.

[![](https://vkrakovna.files.wordpress.com/2021/01/img_2755.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/img_2755.jpg)
We enjoyed large quantities of Portuguese food (as we soon learned, it did not come in small quantities). Our special favorites were local rock mussels called limpets – we ate enough of them to have a shell stacking competition.

[![](https://vkrakovna.files.wordpress.com/2020/12/image.png?w=674)](https://vkrakovna.files.wordpress.com/2020/12/image.png)The winner built a tower of 22, which collapsed before we could take a photo, so here is a tower of 13.
In December we did two short hikes near London, and verified that the basic algorithm of putting Daniel in a car seat and then in a carrier for the hike seems to work pretty well (he mostly sleeps through all this). I’m glad to be able to visit some nature during these strange times, it makes the world feel just a bit less small.

[![](https://vkrakovna.files.wordpress.com/2021/01/20201230_133242.jpg?w=1024)](https://vkrakovna.files.wordpress.com/2021/01/20201230_133242.jpg)Iron age hill fort in Epping Forest
#### 2020 prediction outcomes

Unsurprisingly, some predictions for the past year were messed up by the pandemic.

*Resolutions:*

* Author or coauthor three or more academic papers (3 last year) (70%) – yes (3 papers)
* At most 12 non-research work commitments, such as speaking and organizing (10 last year) (80%) – yes (5 commitments). Easy, since a lot of events got canceled.
* ~~Meditate on at least 270 days (290 last year)~~ (80%) – no (244 days). The past month I only managed to meditate on 9 days, and this has not been good for me, so I need to do better next year.
* Read at least 7 books (5 last year) (70%) – yes (9 books). Human Compatible, The Precipice, Secret of our Success, Ender’s Game, In the Realm of Hungry Ghosts, Watching the English, Positive Birth Book, The Gardener and the Carpenter, Raising a Secure Child.
* ~~At least 700 deep work hours (551 last year)~~ (70%) – no (366 hours). I found it much harder to do deep work at home, and was on parental leave for the last 2.5 months of the year.

*Predictions:*

* ~~I will write at least 5 blog posts~~ (60%) – no (3 posts)
* ~~Eating window at most 11 hours on at least 240 days~~ (228 last year) (70%) – no (131 days), since I stopped doing intermittent fasting this year
* ~~I will visit at least 4 new cities with population over 100,000~~ (11 last year) (70%) – no (2 cities, Birmingham and Funchal). Much less travel than normal this year.
* ~~At most 1 housemate turnover at Deep End~~ (70%) – no (2 housemates). One housemate moved to live with parents in the countryside who would have likely stayed in London under normal circumstances.
* ~~I finish a language in Duolingo~~ (60%) – no, though made some progress on Mandarin (completed checkpoint 2)

#### 2021 resolutions and predictions

*Resolutions:*

* Avoid catching covid (90%)
* Author or coauthor three or more academic papers (3 last year) (70%)
* At most 7 non-research work commitments (5 last year) (80%)
* Meditate on at least 230 days (244 last year) (70%)
* At least 450 deep work hours (366 last year) (70%)
* Do 4 consecutive chinups (70%)

Predictions:

* I will write at least 3 blog posts (3 last year) (60%)
* Janos and I will get vaccinated for covid by the end of June (60%)
* Daniel will get to meet all of his grandparents in person in 2021 (70%)
* I will return to avoiding processed sugar by the end of the year (60%)
* I will finish Hungarian Duolingo (complete checkpoint 5) (70%)

Past new year reviews: [2019-20](https://vkrakovna.wordpress.com/2020/01/09/2019-20-new-year-review/), [2018-19](https://vkrakovna.wordpress.com/2019/01/01/2018-19-new-year-review/), [2017-18](https://vkrakovna.wordpress.com/2018/01/07/2017-18-new-year-review/), [2016-17](https://vkrakovna.wordpress.com/2017/01/09/2016-17-new-year-review/), [2015-16](https://vkrakovna.wordpress.com/2015/12/31/2015-16-new-year-review/), [2014-15](https://vkrakovna.wordpress.com/2015/01/11/2014-15-new-year-review/).

2020-21 New Year review


Tradeoff between desirable properties for baseline choices in impact measures

*(Cross-posted to the [Alignment Forum](https://www.alignmentforum.org/posts/nLhfRpDutEdgr6PKe/tradeoff-between-desirable-properties-for-baseline-choices). Summarized in [Alignment Newsletter #108](https://mailchi.mp/05518aad6baf/an-108why-we-should-scrutinize-arguments-for-ai-risk). Thanks to Carroll Wainwright, Stuart Armstrong, Rohin Shah and Alex Turner for helpful feedback on this post.)*

[**Impact measures**](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-side-effects-e1ac80ea6107)are auxiliary rewards for low impact on the agent’s environment, used to address the problems of side effects and instrumental convergence. A key component of an impact measure is a choice of baseline state: a reference point relative to which impact is measured. Commonly used baselines are the starting state, the initial inaction baseline (the counterfactual where the agent does nothing since the start of the episode) and the stepwise inaction baseline (the counterfactual where the agent does nothing instead of its last action). The **stepwise inaction baseline** is currently considered the best choice because it does not create the following bad incentives for the agent: interference with environment processes or offsetting its own actions towards the objective. This post will discuss a fundamental problem with the stepwise inaction baseline that stems from a **tradeoff** between different desirable properties for baseline choices, and some possible alternatives for resolving this tradeoff.

![](https://lh4.googleusercontent.com/cZnQduT6hcXIABrr6Frv1wMi--O5rg3mgqmcSlXG5d24Vq1a6hTPs2usC_wVJpizOTH5ybd4mh8VgybqncN59YLoEUUlfqlLKohloY-e17EBeTYgVdYg1Tnd6deZnyVHU6ozAL8qCw)
One clearly desirable property for a baseline choice is to effectively penalize high-impact effects, including **delayed effects**. It is well-known that the simplest form of the stepwise inaction baseline does not effectively capture delayed effects. For example, if the agent drops a vase from a high-rise building, then by the time the vase reaches the ground and breaks, the broken vase will be the default outcome. Thus, in order to penalize delayed effects, the stepwise inaction baseline is usually used in conjunction with [inaction rollouts](https://arxiv.org/abs/1902.09725), which predict future outcomes of the inaction policy. Inaction rollouts from the current state and the stepwise baseline state are compared to identify delayed effects of the agent’s actions. In the above example, the current state contains a vase in the air, so in the inaction rollout from the current state the vase will eventually reach the ground and break, while in the inaction rollout from the stepwise baseline state the vase remains intact.

While inaction rollouts are useful for penalizing delayed effects, they do not address all types of delayed effects. In particular, if the task requires setting up a delayed effect, an agent with the stepwise inaction baseline will have no incentive to **undo the delayed effect**. Here are some toy examples that illustrate this problem.

**Door example.** Suppose the agent’s task is to go to the store, which requires opening the door in order to leave the house. Once the door has been opened, the effects of opening the door are part of the stepwise inaction baseline, so the agent has no incentive to close the door as it leaves.

**Red light example.**  Suppose the agent’s task is to drive from point A to point B along a straight road, with a reward for reaching point B. To move towards point B, the agent needs to accelerate. Once the agent has accelerated, it travels at a constant speed by default, so the noop action will move the agent along the road towards point B. Along the road (s1), there is a red light and a pedestrian crossing the road. The noop action in s1 crosses the red light and hits the pedestrian (s2). To avoid this, the agent needs to deviate from the inaction policy by stopping (s4) and then accelerating (s5).

![](https://lh3.googleusercontent.com/XyQQ2j2yFhVm5xJ7VBF39OL32AdWRPQt2tjncx7JHqJlw_ElC9vnQ26HsNodGHQDx1QTxBoFRIswkK6YY3eolgC45C8j3hyE-AZqLVyJ8g0IvlfEvyUWDHvkT6qAp1JTEls8R33tGw)
The stepwise inaction baseline will incentivize the agent to run the red light and go to s3. The inaction rollout at s0 penalizes the agent for the predicted delayed effect of running over the pedestrian when it takes the accelerating action to go to s1. The agent receives this penalty whether or not it actually ends up running the red light or not. Once the agent has reached s1, running the red light becomes the default outcome, so the agent is not penalized for doing so (and would likely be penalized for stopping). Thus, the stepwise inaction baseline gives no incentive to avoid running the red light, while the initial inaction baseline compares to s0 and thus incentivizes the agent to stop at the red light.

This problem with the stepwise baseline arises from a tradeoff between penalizing delayed effects and avoiding **offsetting** incentives. The stepwise structure that makes it effective at avoiding offsetting makes it less effective at penalizing delayed effects. While delayed effects are undesirable, undoing the agent’s actions is not necessarily bad. In the red light example, the action of stopping at the red light is offsetting the accelerating action. Thus, offsetting can be necessary for avoiding delayed effects while completing the task.

Whether offsetting an effect is desirable depends on whether this effect is **part of the task objective**. In the door-opening example, the action of opening the door is instrumental for going to the store, and many of its effects (e.g. strangers entering the house through the open door) are not part of the objective, so it is desirable for the agent to undo this action. In the **vase environment** shown below, the task objective is to prevent the vase from falling off the end of the belt and breaking, and the agent is rewarded for taking the vase off the belt. The effects of taking the vase off the belt are part of the objective, so it is undesirable for the agent to undo this action.

![](https://lh5.googleusercontent.com/I9s_s5dOcFnvBPD3wgrPLabzAqah2xeqllexP40fzEHCtSCh_swVv___qtYpT1vaNHj1T-oGGzoOjQ7is6ltjO5Qnshe5emZ4Gps2qv_cUSY02T2pHgQz8IqKDjnKM5Mt7wDXTTqBQ)Source: [Designing agent incentives to avoid side effects](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-side-effects-e1ac80ea6107)
The difficulty of identifying these “task effects” that are part of the objective creates a **tradeoff** between penalizing delayed effects and avoiding undesirable offsetting. This tradeoff can be avoided by the starting state baseline, which however produces interference incentives. The stepwise inaction baseline cannot resolve the tradeoff, since it avoids all types of offsetting, including desirable offsetting.

**The initial inaction baseline can resolve this tradeoff** by allowing offsetting and relying on the task reward to capture task effects and penalize the agent for offsetting them. While we cannot expect the task reward to capture what the agent should not do (unnecessary impact), capturing task effects falls under what the agent should do, so it seems reasonable to rely on the reward function for this. This would work similarly to the impact penalty penalizing all impact, and the task reward compensating for this in the case of impact that’s needed to complete the task.

This can be achieved using a state-based reward function that assigns reward to all states where the task is completed. For example, in the vase environment, a state-based reward of 1 for states with an intact vase (or with vase off the belt) and 0 otherwise would remove the offsetting incentive.

If it is not feasible to use a reward function that penalizes offsetting task effects, the initial inaction baseline could be **modified** to avoid this kind of offsetting. If we assume that the task reward is sparse and doesn’t include shaping terms, we can reset the initial state for the baseline whenever the agent receives a task reward (e.g. the reward for taking the vase off the belt in the vase environment). This results in a kind of hybrid between initial and stepwise inaction. To ensure that this hybrid baseline effectively penalizes delayed effects, we still need to use inaction rollouts at the reset and terminal states.

Another desirable property of the stepwise inaction baseline is **the Markov property**: it can be computed based on the previous state, independently of the path taken to that state. The initial inaction baseline is not Markovian, since it compares to the state in the initial rollout at the same time step, which requires knowing how many time steps have passed since the beginning of the episode. We could modify the initial inaction baseline to make it Markovian, e.g. by sampling a single baseline state from the inaction rollout from the initial state, or by only computing a single penalty at the initial state by comparing an agent policy rollout with the inaction rollout.

To summarize, we want a baseline to satisfy the following **desirable properties:** penalizing delayed effects, avoiding interference incentives, and the Markov property. We can consider avoiding offsetting incentives for task effects as a desirable property for the task reward, rather than the baseline. Assuming such a well-specified task reward, a Markovian version of the initial inaction baseline can satisfy all the criteria.


Possible takeaways from the coronavirus pandemic for slow AI takeoff

*(Cross-posted to [LessWrong](https://www.lesswrong.com/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai). *Summarized in [Alignment Newsletter #104](https://mailchi.mp/ba4d1765368f/an-104-the-perils-of-inaccessible-information-and-what-we-can-learn-about-ai-alignment-from-covid).*Thanks to Janos Kramar for helpful feedback on this post.)*

As the covid-19 pandemic unfolds, we can draw lessons from it for managing future global risks, such as other pandemics, climate change, and risks from advanced AI. In this post, I will focus on possible implications for AI risk. For a broader treatment of this question, I recommend [FLI’s covid-19 page](https://futureoflife.org/covid-19/) that includes expert interviews on the implications of the pandemic for other types of risks.

A key element in AI risk scenarios is the speed of takeoff – whether advanced AI is developed gradually or suddenly. Paul Christiano’s post on [takeoff speeds](https://sideways-view.com/2018/02/24/takeoff-speeds/) defines slow takeoff in terms of the economic impact of AI as follows: “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.” It argues that slow AI takeoff is more likely than fast takeoff, but is not necessarily easier to manage, since it poses different challenges, such as large-scale coordination. This post expands on this point by examining some parallels between the coronavirus pandemic and a slow takeoff scenario. The upsides of slow takeoff include the ability to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. I would argue that the covid-19 pandemic had these properties, but most of the world’s institutions did not take advantage of them. This suggests that, unless our institutions improve, we should not expect the slow AI takeoff scenario to have a good default outcome.

1. **Learning from experience.** In the slow takeoff scenario, general AI is expected to appear in a world that has already experienced transformative change from less advanced AI, and institutions will have a chance to learn from problems with these AI systems. An analogy could be made with learning from dealing with less “advanced” epidemics like SARS that were not as successful as covid-19 at spreading across the world. While some useful lessons were learned, they were not successfully generalized to covid-19, which had somewhat different properties than these previous pathogens (such as asymptomatic transmission and higher virulence). Similarly, general AI may have somewhat different properties from less advanced AI that would make mitigation strategies more difficult to generalize.
2. **Warning signs.** In the coronavirus pandemic response, there has been a lot of variance in how successfully governments acted on warning signs. Western countries had at least a month of warning while the epidemic was spreading in China, which they could have used to stock up on PPE and build up testing capacity, but most did not do so. Experts have [warned](https://www.mdpi.com/1999-4915/11/3/210?fbclid=IwAR1in9B-qMWRMfHNvFoD5dheRQaXmWtFZ41UDnXfcB0GyfKc999NlhlhW3U) about the likelihood of a coronavirus outbreak for many years, but this did not lead most governments to stock up on medical supplies. This was a failure to take cheap preventative measures in response to advance warnings about a widely recognized risk with tangible consequences, which is not a good sign for the case where the risk is less tangible and well-understood (such as risk from general AI).
3. **Consensus on the problem.** During the covid-19 epidemic, the abundance of warning signs and past experience with previous pandemics created an opportunity for a timely consensus that there is a serious problem. However, it actually took a long time for a broad consensus to emerge – the virus was [often](https://www.theguardian.com/world/2020/mar/15/wisconsin-midwest-coronavirus-response) [dismissed](https://www.sciencealert.com/infectious-disease-researcher-explains-why-the-coronavirus-is-bringing-out-the-worst-in-us) as “overblown” and “just like the flu” as late as March 2020. A timely response to the risk required acting before there was a consensus, thus risking the appearance of overreacting to the problem. I think we can also expect this to happen with advanced AI. Similarly to the discussion of covid-19, there is an unfortunate irony where those who take a dismissive position on advanced AI risks are often seen as cautious, prudent skeptics, while those who advocate early action are portrayed as “panicking” and overreacting. The “moving goalposts” effect, where new advances in AI are dismissed as not real AI, could continue indefinitely as increasingly advanced AI systems are deployed. I would expect the [“no fire alarm” hypothesis](https://intelligence.org/2017/10/13/fire-alarm/) to hold in the slow takeoff scenario – there may not be a consensus on the importance of general AI until it arrives, so risks from advanced AI would continue to be seen as “overblown” until it is too late to address them.

We can hope that the transformative technological change involved in the slow takeoff scenario will also help create more competent institutions without these weaknesses. We might expect that institutions unable to adapt to the fast pace of change will be replaced by more competent ones. However, we could also see an increasingly chaotic world where institutions fail to adapt without better institutions being formed quickly enough to replace them. Success in the slow takeoff scenario depends on institutional competence and large-scale coordination. Unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff.


2019-20 New Year review

This is an annual post reviewing the last year and making resolutions and predictions for next year. This year’s edition features sleep tracking, intermittent fasting, overcommitment busting, and evaluating calibration for all annual predictions since 2014.

**2019 review**

*AI safety research:*

* Wrote an [updated version](https://arxiv.org/abs/1806.01186) of the relative reachability paper, including an ablation study on design choices.
* Coauthored a paper on [modeling AGI Safety frameworks with causal influence diagrams](https://arxiv.org/abs/1906.08663), accepted to IJCAI workshop.
* Wrote a paper on [avoiding side effects by considering future tasks](https://drive.google.com/open?id=0B3mY6u_lryzdMGpEbEljRmFIS2hZWno1clExMDRuVVZWMnJV), accepted to NeurIPS workshop.
* Co-ran a subteam of the safety team focusing on agent incentive design (ongoing).

*AI safety outreach:*

* Co-organized FLI’s [Beneficial AGI conference](https://futureoflife.org/beneficial-agi-2019/) in Puerto Rico, a more long-term focused sequel to the original Puerto Rico conference and the Asilomar conference. This year I was the program chair for the technical safety track of the conference.
* Co-organized the ICLR AI safety workshop, [Safe Machine Learning: Specification, Robustness and Assurance](https://sites.google.com/corp/view/safeml-iclr2019/). This was my first time running a paper reviewing process.
* Gave a [talk](https://drive.google.com/file/d/1wTsM75MZaRvCoiCeQnm_8Tln5RJxCqhO/view) at the IJCAI AI safety workshop on specification, robustness an assurance problems.
* Took part in the DeepMind podcast [episode](https://deepmind.com/blog/article/podcast-episode-4-ai-robot) on AI safety (“I, robot”).

*Work effectiveness:*

* At the beginning of the year, I found myself overcommitted and kind of burned out. My previous efforts to reduce overcommitment had proved insufficient to not feel stressed and overwhelmed most of the time.
* In February, I made a rule for myself to decline all non-research commitments that don’t seem like exceptional opportunities. The form that I made last year for evaluating commitments (which I have to fill out before accepting anything) has been helpful for enforcing this rule and avoiding impulsive decisions. The number of commitments went down from 24 in 2018 to 10 in 2019. This has been working well in terms of having more time for research and feeling better about life.
* Organizing a conference and a workshop back to back was a bit much, and I feel done with organizing large events for a while.

![and-now-my-watch-has-ended](https://vkrakovna.files.wordpress.com/2020/01/and-now-my-watch-has-ended.jpg?w=300)

* Stopped using work cycles and pomodoros since I’ve been finding the structure a bit too restrictive. Might resume at some point.
* Hours of “deep work” per month, as defined in the Deep Work book. This includes things like thinking about research problems, coding, reading and writing papers. It does not include email, organizing, most meetings, coding logistics (e.g. setup or running experiments), etc.

![deep_work_2019](https://vkrakovna.files.wordpress.com/2020/01/deep_work_2019.png)

* For comparison, deep work hours from 2018. My definition of deep work has somewhat broadened over time, but not enough to account for this difference.![deep_work_2018](https://vkrakovna.files.wordpress.com/2020/01/deep_work_2018.png)

*Health / self-care:*

* I had 7 colds this past year, which is a lot more than my usual rate of 1-2 per year. The first three were in Jan-Feb, which seemed related to the overcommitment burnout. Hopefully supplementing zinc will help.
* Averaged 7.2 hours of sleep per night, excluding jetlag (compared to 6.9 hours in 2018).
* About a 10% rate of insomnia (excluding jetlag), similar to the end of last year.
* Tried the Oura ring and the Dreem headband for measuring sleep quality. The Oura ring consistently thinks I wake up many times per night (probably because I move around a lot) and predicts less than half an hour each of deep and REM sleep. The Dreem, which actually measures EEG signals, estimates that I get an average of 1.3 hours of deep sleep and 1.8 hours of REM sleep per night, which is more than I expected.
* Started a relaxed form of intermittent fasting in March (aiming for a 10 hour eating window), mostly for longevity and to improve my circadian rhythm. My average eating window length over the year was 10.5 hours, so I wasn’t very strict about it (mostly just avoiding snacks after dinner). One surprising thing I learned was that I can fall asleep just fine while hungry, and am often less hungry when I wake up. My average hours of sleep went up from 6.96 in the 6 months before starting intermittent fasting to 7.32 in the 6 months after. I went to sleep 44 minutes earlier and waking up 20 minutes earlier on average, though the variance of my bedtime actually went up a bit. Overall it seems plausibly useful easy enough to continue next year.

*Fun stuff:*

* Did a Caucasus hiking trek in Georgia with family, and consumed a lot of wild berries and hazelnuts along the way.

![vika_berries](https://vkrakovna.files.wordpress.com/2020/01/vika_berries.jpg?w=1024)

* Did a road trip in southern Iceland (also with family), saw a ridiculous number of pretty waterfalls, and was in the same room with (artificial) lava.

![waterfall](https://vkrakovna.files.wordpress.com/2020/01/waterfall.jpg)

* Took an advanced class in aerial silks for the first time (I felt a bit underqualified, but learned a lot of fun moves).
* Ran a half-marathon along the coast in Devon on hilly terrain in 3 hours and 23 minutes.
* Made some progress on handstands in yoga class (can hold it away from the wall for a few seconds)
* Did two circling retreats (relational meditation)
* Read books: The Divide, 21 Lessons for the 21st Century, The Circadian Code, So Good They Can’t Ignore You, Ending Aging (skimmed).
* Got into Duolingo (brushed up my Spanish and learned a bit of Mandarin). Currently in a quasi-competition with Janos for studying each other’s languages.

#### 2019 prediction outcomes

*Resolutions:*

1. Author or coauthor two or more academic papers (50%) – yes (3 papers)
2. Accept at most 17 non-research commitments (24 last year) (60%) – yes (10 commitments)
3. Meditate on at least 250 days (60%) – yes (290 days)

*Predictions:*

1. ~~Relative reachability paper accepted at a major conference, not counting workshops~~ (60%) – no
2. ~~Continue avoiding processed sugar for the next year~~ (85%) – no (still have the intention and mostly follow it, but less strictly / consistently)
3. 1-2 housemate turnover at Deep End (2 last year) (80%) – yes (1 housemate moved in)
4. ~~At least 5 rationality sessions will be hosted at Deep End~~ (80%) – no

*Calibration over all annual reviews:*

* 50-70% well-calibrated, 80-90% overconfident (66 predictions total)
* Calibration is generally better in 2017-19 (23 predictions) than in 2014-16 (43 predictions). There were only 3 70% predictions in 2017-19, so the 100% accuracy is noisy.
* Unsurprisingly, resolutions are more often correct than other predictions (72% vs 56% correct)

![calibration_chart](https://vkrakovna.files.wordpress.com/2019/12/calibration_chart.png)

#### 2020 resolutions and predictions

Resolutions

* Author or coauthor three or more academic papers (3 last year) (70%)
* At most 12 non-research commitments (10 last year) (80%)
* Meditate on at least 270 days (290 last year) (80%)
* Read at least 7 books (5 last year) (70%)
* At least 700 deep work hours (551 last year) (70%)

Predictions

* I will write at least 5 blog posts (60%)
* Eating window at most 11 hours on at least 240 days (228 last year) (70%)
* I will visit at least 4 new cities with population over 100,000 (11 last year) (70%)
* At most 1 housemate turnover at Deep End (70%)
* I finish a language in Duolingo (60%)

Past new year reviews: [2018-19](https://vkrakovna.wordpress.com/2019/01/01/2018-19-new-year-review/), [2017-18](https://vkrakovna.wordpress.com/2018/01/07/2017-18-new-year-review/), [2016-17](https://vkrakovna.wordpress.com/2017/01/09/2016-17-new-year-review/), [2015-16](https://vkrakovna.wordpress.com/2015/12/31/2015-16-new-year-review/), [2014-15](https://vkrakovna.wordpress.com/2015/01/11/2014-15-new-year-review/).

####

2019-20 New Year review


Retrospective on the specification gaming examples list

My [post](https://www.lesswrong.com/posts/AanbbjYr5zckMKde7/specification-gaming-examples-in-ai) about the [specification gaming list](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/) was recently nominated for the [LessWrong 2018 Review](https://www.lesswrong.com/posts/qXwmMkEBLL59NkvYR/the-lesswrong-2018-review-posts-need-at-least-2-nominations) (sort of like a test of time award), which prompted me to write a retrospective ([cross-posted here](https://www.lesswrong.com/posts/AanbbjYr5zckMKde7/specification-gaming-examples-in-ai#33tup73uagNdXJMcD)).

I’ve been pleasantly surprised by how much this resource has caught on in terms of people using it and referring to it (definitely more than I expected when I made it). There were 30 examples on the list when was posted in April 2018, and 20 new examples have been contributed through the form since then.  I think the list has several properties that contributed to wide adoption: it’s fun, standardized, up-to-date, comprehensive, and collaborative.

Some of the appeal is that it’s fun to read about AI cheating at tasks in unexpected ways (I’ve seen a lot of people post on Twitter about their favorite examples from the list). The standardized spreadsheet format seems easier to refer to as well. I think the crowdsourcing aspect is also helpful – this helps keep it current and comprehensive, and people can feel some ownership of the list since can personally contribute to it. My overall takeaway from this is that safety outreach tools are more likely to be impactful if they are fun and easy for people to engage with.

This list had a surprising amount of impact relative to how little work it took me to put it together and maintain it. The hard work of finding and summarizing the examples was done by the people putting together the lists that the master list draws on (Gwern, Lehman, Olsson, Irpan, and others), as well as the people who submit examples through the form. What I do is put them together in a common format and clarify and/or shorten some of the summaries. I also curate the examples to determine whether they fit the definition of specification gaming (as opposed to simply a surprising behavior or solution). Overall, I’ve probably spent around 10 hours so far on creating and maintaining the list, which is not very much. This makes me wonder if there is other low hanging fruit in the safety resources space that we haven’t picked yet.

I have been using it both as an outreach and research tool. On the outreach side, the resource has been helpful for making the argument that safety problems are hard and need general solutions, by making it salient just in how many ways things could go wrong. When presented with an individual example of specification gaming, people often have a default reaction of “well, you can just close the loophole like this”. It’s easier to see that this approach does not scale when presented with 50 examples of gaming behaviors. Any given loophole can seem obvious in hindsight, but 50 loopholes are much less so. I’ve found this useful for communicating a sense of the difficulty and importance of Goodhart’s Law.

On the research side, the examples have been helpful for trying to clarify the distinction between [reward gaming and tampering problems](https://www.lesswrong.com/posts/yXPT4nr4as7JvxLQa/classifying-specification-problems-as-variants-of-goodhart-s). Reward gaming happens when the reward function is designed incorrectly (so the agent is gaming the design specification), while reward tampering happens when the reward function is implemented incorrectly or embedded in the environment (and so can be thought of as gaming the implementation specification). The boat race example is reward gaming, since the score function was defined incorrectly, while the Qbert agent finding a bug that makes the platforms blink and gives the agent millions of points is reward tampering. We don’t currently have any real examples of the agent gaining control of the reward channel (probably because the action spaces of present-day agents are too limited), which seems qualitatively different from the numerous examples of agents exploiting implementation bugs.

I’m curious what people find the list useful for – as a safety outreach tool, a research tool or intuition pump, or something else? I’d also be interested in suggestions for improving the list (formatting, categorizing, etc). Thanks everyone who has contributed to the resource so far!


Classifying specification problems as variants of Goodhart’s Law

*(Coauthored with Ramana Kumar and cross-posted from the [Alignment Forum](https://www.alignmentforum.org/posts/yXPT4nr4as7JvxLQa/classifying-specification-problems-as-variants-of-goodhart-s). Summarized in [Alignment Newsletter #76](https://mailchi.mp/1106d0ce6766/an-76how-dataset-size-affects-robustness-and-benchmarking-safe-exploration-by-measuring-constraint-violations).)*

There are a few different classifications of safety problems, including the [Specification, Robustness and Assurance (SRA) taxonomy](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) and the [Goodhart’s Law taxonomy](https://www.alignmentforum.org/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy). In SRA, the specification category is about defining the purpose of the system, i.e. specifying its incentives.  Since incentive problems can be seen as manifestations of Goodhart’s Law, we explore how the specification category of the SRA taxonomy maps to the Goodhart taxonomy. The mapping is an attempt to integrate different breakdowns of the safety problem space into a coherent whole. We hope that a consistent classification of current safety problems will help develop solutions that are effective for entire classes of problems, including future problems that have not yet been identified.

The SRA taxonomy defines three different types of specifications of the agent’s objective: ideal (a perfect description of the wishes of the human designer), design (the stated objective of the agent) and revealed (the objective recovered from the agent’s behavior). It then divides specification problems into **design** problems (e.g. [side effects](https://arxiv.org/abs/1806.01186)) that correspond to a difference between the ideal and design specifications, and **emergent** problems (e.g. [tampering](https://arxiv.org/abs/1908.04734)) that correspond to a difference between the design and revealed specifications.

In the Goodhart taxonomy, there is a variable U\* representing the true objective, and a variable U representing the proxy for the objective (e.g. a reward function). The taxonomy identifies four types of Goodhart effects: **regressional** (maximizing U also selects for the difference between U and U\*), **extremal** (maximizing U takes the agent outside the region where U and U\* are correlated), **causal** (the agent intervenes to maximize U in a way that does not affect U\*), and **adversarial** (the agent has a different goal W and exploits the proxy U to maximize W).

We think there is a correspondence between these taxonomies: design problems are regressional and extremal Goodhart effects, while emergent problems are causal Goodhart effects. The rest of this post will explain and refine this correspondence.

![sra-goodhart](https://vkrakovna.files.wordpress.com/2019/08/sra-goodhart.png?w=625)

The SRA taxonomy needs to be refined in order to capture the distinction between regressional and extremal Goodhart effects, and to pinpoint the source of causal Goodhart effects. To this end, we add a **model** specification as an intermediate point between the ideal and design specifications, and an **implementation** specification between the design and revealed specifications.

The **model** specification is the best proxy within a chosen formalism (e.g. model class or specification language), i.e. the proxy that most closely approximates the ideal specification. In a reinforcement learning setting, the model specification is the reward function (defined in the given MDP/R over the given state space) that best captures the human designer’s preferences.

* The ideal-model gap corresponds to the **model design** problem (regressional Goodhart): choosing a model that is tractable but also expressive enough to approximate the ideal specification well.
* The model-design gap corresponds to **proxy design** problems (extremal Goodhart), such as [specification gaming](http://tinyurl.com/specification-gaming) and side effects.

While the design specification is a high-level description of what should be executed by the system, the **implementation** specification is a specification that can be executed, which includes agent and environment code (e.g. an executable Linux binary). (We note that it is also possible to define other specification levels at intermediate levels of abstraction between design and implementation, e.g. using pseudocode rather than executable code.)

* The design-implementation gap corresponds to **tampering** problems (causal Goodhart), since they exploit implementation flaws (such as bugs that allow the agent to overwrite the reward). (Note that tampering problems are referred to as wireheading and delusions in the SRA.)
* The implementation-revealed gap corresponds to robustness problems in the SRA (e.g. unsafe exploration).

![sra-goodhart-refined](https://vkrakovna.files.wordpress.com/2019/08/sra-goodhart-refined.png?w=625)

In the **model design** problem, U is the best approximation of U\* within the given model.  As long as the global maximum M for U is not exactly the same as the global maximum M\* for U\*, the agent will not find M\*. This corresponds to regressional Goodhart: selecting for U will also select for the difference between U and U\*, so the optimization process will overfit to U at the expense of U\*.

![model-design](https://vkrakovna.files.wordpress.com/2019/08/model-design-e1566233809558.png?w=350&h=174)

In **proxy** **design** problems, U and U\* are correlated under normal circumstances, but the correlation breaks in situations when U is maximized, which is an extremal Goodhart effect. The proxy U is often designed to approximate U\* by having a maximum at a global maximum M\* of U\*. Different ways that this approximation fails produce different problems.

* In specification gaming problems, M\* turns out to be a local (rather than global) maximum for U, e.g. if M\* is the strategy of following the racetrack in the boat race game. The agent finds the global maximum M for U, e.g. the strategy of going in circles and repeatedly hitting the same reward blocks. This is an extrapolation of the reward function outside the training domain that it was designed for, so the correlation with the true objective no longer holds. This is an extremal Goodhart effect due to regime change.

![proxy-design-spec-gaming](https://vkrakovna.files.wordpress.com/2019/08/proxy-design-spec-gaming-e1566233862970.png?w=485&h=163)

* In side effect problems, M\* is a global maximum for U, but U incorrectly approximates U\* by being flat in certain dimensions (corresponding to indifference to certain variables, e.g. whether a vase is broken). Then the set of global maxima for U is much larger than the set of global maxima for U\*, and most points in that set are not global maxima for U\*. Maximizing U can take the agent into a region where U doesn’t match U\*, and the agent finds a point M that is also a global maximum for U, but not a global maximum for U\*. This is an extremal Goodhart effect due to model insufficiency.

![proxy-design-side-effects](https://vkrakovna.files.wordpress.com/2019/08/proxy-design-side-effects-e1566233840931.png?w=388&h=144)

Current solutions to proxy design problems involve taking the proxy less literally: by injecting uncertainty (e.g. quantilization), avoiding extrapolation (e.g. inverse reward design), or adding a term for omitted preferences (e.g. impact measures).

In **tampering** problems, we have a causal link U\* -> U. Tampering occurs when the agent intervenes on some variable W that has a causal effect on U that does not involve U\*, which is a causal Goodhart effect. W could be the reward function parameters, the human feedback data (in reward learning), the observation function parameters (in a POMDP), or the status of the shutdown button. The overall structure is  U\* -> U <- W.

For example, in the [Rocks & Diamonds environment](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd), U\* is the number of diamonds delivered by the agent to the goal area. Intervening on the reward function to make it reward rocks increases the reward U without increasing U\* (the number of diamonds delivered).

Current solutions to tampering problems involve modifying the causal graph to remove the tampering incentives, e.g. by using [approval-direction](https://ai-alignment.com/model-free-decisions-6e6609f5d99e) or introducing [counterfactual variables](https://arxiv.org/abs/1711.05541).

[Updated] We think that [mesa-optimization](https://arxiv.org/abs/1906.01820) belongs in the implementation-revealed gap, rather than in the design-implementation gap, since it can happen during the learning process even if the implementation specification matches the ideal specification, and can be seen as a [robustness problem](https://www.alignmentforum.org/posts/2mhFMgtAjFJesaSYR/2-d-robustness). When we consider this problem one level down, as a specification problem for the mesa-optimizer from the main agent’s perspective, it can take the form of any of the four Goodhart effects. The four types of alignment problems in the mesa-optimization paper can be mapped to the four types of Goodhart’s Law as follows: approximate alignment is regressional, side effect alignment is extremal, instrumental alignment is causal, and deceptive alignment is adversarial.

This correspondence is consistent with the [connection](https://www.alignmentforum.org/posts/zdeYiQgwYRs2bEmCK/applying-overoptimization-to-selection-vs-control-optimizing) between the Goodhart taxonomy and the **selection vs control** distinction, where regressional and extremal Goodhart are more relevant for selection, while causal Goodhart is more relevant for control. The design specification is generated by a selection process, while the revealed specification is generated by a control process. Thus, design problems represent difficulties with selection, while emergent problems represent difficulties with control.

[Updated] Putting it all together:

![sra-goodhart-all1](https://vkrakovna.files.wordpress.com/2019/08/sra-goodhart-all1.png?w=625)

In terms of the limitations of this mapping, we are not sure about model specification being the dividing line between regressional and extremal Goodhart. For example, a poor choice of model specification could deviate from the ideal specification in systematic ways that result in extremal Goodhart effects. It is also unclear how adversarial Goodhart fits into this mapping. Since an adversary can exploit any differences between U\* and U (taking advantage of the other three types of Goodhart effects) it seems that adversarial Goodhart effects can happen anywhere in the ideal-implementation gap.

We hope that you find the mapping useful for your thinking about the safety problem space, and welcome your feedback and comments. We are particularly interested if you think some of the correspondences in this post are wrong.

*(Thanks to Jan Leike and Tom Everitt for their helpful feedback on this post.)*


ICLR Safe ML workshop report

This year the ICLR conference hosted topic-based workshops for the first time (as opposed to a single track for workshop papers), and I co-organized the [Safe ML workshop](https://sites.google.com/corp/view/safeml-iclr2019/home?authuser=0). One of the main goals was to bring together near and long term safety research communities.

![near-long-term](https://vkrakovna.files.wordpress.com/2019/06/near-long-term.png?w=625)

The workshop was structured according to a [taxonomy](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) that incorporates both near and long term safety research into three areas – specification, robustness and assurance.

|  |  |  |
| --- | --- | --- |
| **Specification**: define the purpose of the system | **Robustness**: design system to withstand perturbations | **Assurance**: monitor and control system activity |
| * Reward hacking
* Side effects
* Preference learning
* Fairness
 | * Adaptation
* Verification
* Worst-case robustness
* Safe exploration
 | * Interpretability
* Monitoring
* Privacy
* Interruptibility
 |

We had an invited talk and a contributed talk in each of the three areas.

#### Talks

In the **specification** area, Dylan Hadfield-Menell spoke about [formalizing the value alignment problem](https://slideslive.com/38915783/formalizing-the-value-alignment-problem-in-ai) in the Inverse RL framework.

![formalizing-value-alignment](https://vkrakovna.files.wordpress.com/2019/06/formalizing-value-alignment.png?w=625)

David Krueger [presented](https://slideslive.com/38915784/misleading-metaobjectives-and-hidden-incentives-for-distributional-shift) a [paper](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) on hidden incentives for the agent to shift its task distribution in the meta-learning setting.

![distributional-shift](https://vkrakovna.files.wordpress.com/2019/06/distributional-shift.png?w=300&h=217)

In the **robustness** area, Ian Goodfellow argued for [dynamic defenses against adversarial examples](https://slideslive.com/38915790/the-case-for-dynamic-defenses-against-adversarial-examples) and encouraged the research community to consider threat models beyond small perturbations within a norm ball of the original data point.

![threat-models](https://vkrakovna.files.wordpress.com/2019/06/threat-models.png?w=625)

Avraham Ruderman [presented](https://slideslive.com/38915789/uncovering-surprising-behaviors-in-reinforcement-learning-via-worstcase-analysis) a [paper](https://drive.google.com/uc?export=download&id=1z_d1EjvKWlh2L39xxGXYDWP1wQwnlNLQ) on worst-case analysis for discovering surprising behaviors (e.g. failing to find the goal in simple mazes).

![mazes](https://vkrakovna.files.wordpress.com/2019/06/mazes.png?w=625)

In the **assurance** area, Cynthia Rudin argued that interpretability doesn’t have to trade off with accuracy (especially in applications), and that it is helpful for solving research problems in all areas of safety.

![interpretability-taxonomy](https://vkrakovna.files.wordpress.com/2019/06/interpretability-taxonomy.png?w=625)

Beomsu Kim [presented](https://slideslive.com/38915785/bridging-adversarial-robustness-and-gradient-interpretability) a [paper](https://drive.google.com/uc?export=download&id=1FUlKR07jf1VQb6M-4VVTxxu3oDkK0x-8) explaining why adversarial training improves the interpretability of gradients for deep neural networks.

![gradient-interpretability](https://vkrakovna.files.wordpress.com/2019/06/gradient-interpretability.png?w=625)

#### Panels

The workshop panels discussed possible overlaps between different research areas in safety and research priorities going forward.

In terms of **overlaps**, the main takeaway was that advancing interpretability is useful for all safety problems. Also, adversarial robustness can contribute to value alignment – e.g. reward gaming behaviors can be viewed as a system finding adversarial examples for its reward function. However, there was a cautionary point that while near- and long-term problems are often similar, solutions might not transfer well between these areas (e.g. some solutions to near-term problems might not be sufficiently general to help with value alignment).

![panel](https://vkrakovna.files.wordpress.com/2019/06/panel.jpg?w=625)

The **research priorities** panel recommended more work on adversarial examples with realistic threat models (as mentioned above), complex environments for testing value alignment (e.g. creating new structures in Minecraft without touching existing ones), fairness formalizations with more input from social scientists, and improving cybersecurity.

#### Papers

Out of the 35 accepted papers, 5 were on long-term safety / value alignment, and the rest were on near-term safety. Half of the near-term paper submissions were on adversarial examples, so the resulting pool of accepted papers was skewed as well: 14 on adversarial examples, 5 on interpretability, 3 on safe RL, 3 on other robustness, 2 on fairness, 2 on verification, and 1 on privacy. Here is a summary of the value alignment papers:

[Misleading meta-objectives and hidden incentives for distributional shift](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) by Krueger et al shows that RL agents in a meta-learning context have an incentive to shift their task distribution instead of solving the intended task. For example, a household robot whose task is to predict whether its owner will want coffee could wake up its owner early in the morning to make this prediction task easier. This is called a ‘self-induced distributional shift’ (SIDS), and the incentive to do so is a ‘hidden incentive for distributional shift’ (HIDS). The paper demonstrates this behavior experimentally and shows how to avoid it.

![coffee-robot](https://vkrakovna.files.wordpress.com/2019/06/coffee-robot.png?w=625)

[How useful is quantilization for mitigating specification-gaming?](https://drive.google.com/uc?export=download&id=13qAfOm8McRvXS33MCNH0ia4ApMIClZP9) by Ryan Carey introduces variants of several classic environments (Mountain Car, Hopper and Video Pinball) where the observed reward differs from the true reward, creating an opportunity for the agent to game the specification of the observed reward. The paper shows that a quantilizing agent avoids specification gaming and performs better in terms of true reward than both imitation learning and a regular RL agent on all the environments.

![spec-gaming](https://vkrakovna.files.wordpress.com/2019/06/spec-gaming.png?w=625)

[Delegative Reinforcement Learning: learning to avoid traps with a little help](https://drive.google.com/uc?export=download&id=1xa7UpGGODl6mszNWkA4XQGPyeopsNuWu) by Vanessa Kosoy introduces an RL algorithm that avoids traps in the environment (states where regret is linear) by delegating some actions to an external advisor, and achieves sublinear regret in a continual learning setting. (Summarized in [Alignment Newsletter #57](http://eepurl.com/gtZ8DD))

![trap](https://vkrakovna.files.wordpress.com/2019/06/trap.gif?w=625)

[Generalizing from a few environments in safety-critical reinforcement learning](https://drive.google.com/uc?export=download&id=1Q_lGKZzJwc7h2f8oYgZDBAmRttMBkgCk) by Kenton et al investigates how well RL agents avoid catastrophes in new gridworld environments depending on the number of training environments. They find that both model ensembling and learning a catastrophe classifier (used to block actions) are helpful for avoiding catastrophes, with different safety-performance tradeoffs on new environments.

![dist-shift-catastrophes](https://vkrakovna.files.wordpress.com/2019/06/dist-shift-catastrophes.png?w=625)

[Regulatory markets for AI safety](https://drive.google.com/uc?export=download&id=1bFPiwLrZc7SQTMg2_bW4gt0PaS5NyqOH) by Clark and Hadfield proposes a new model for regulating AI development where regulation targets are required to choose regulatory services from a private market that is overseen by the government. This allows regulation to efficiently operate on a global scale and keep up with the pace of technological development and better ensure safe deployment of AI systems. (Summarized in [Alignment Newsletter #55](http://eepurl.com/gp5MFP))

![regulatory-markets](https://vkrakovna.files.wordpress.com/2019/06/regulatory-markets.png?w=625)

The workshop got a pretty good turnout (around 100 people). Thanks everyone for participating, and thanks to our reviewers, sponsors, and my fellow organizers for making it happen!

![audience](https://vkrakovna.files.wordpress.com/2019/06/audience.jpg?w=625)

*(Cross-posted to the [FLI blog](https://futureoflife.org/2019/06/18/iclr-safe-ml-workshop-report/). Thanks to Janos Kramar for his feedback on this post.)*

ICLR Safe ML workshop report


2018-19 New Year review

#### 2018 progress

Research / AI safety:

* Wrote a paper on [measuring side effects using relative reachability](https://arxiv.org/abs/1806.01186) in May, and presented the results at the ICML GoalsRL workshop and the AI safety summer school. Since then, some [new](https://openreview.net/forum?id=rkevMnRqYQ) [approaches](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) have come out using my method as a baseline :).
* Made a list of 30 [specification gaming examples in AI](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/) (assembled from several existing lists). Since the list was posted in April, 16 new examples have been contributed through the form (thanks everyone!). The list received [some attention on Twitter](https://twitter.com/mogwai_poet/status/1060286856493813760), and I was interviewed about it by [Wired](https://www.wired.com/story/when-bots-teach-themselves-to-cheat/) and the [Times](https://www.thetimes.co.uk/edition/news/why-ai-can-be-too-clever-for-its-own-good-pvlgzrzbp).
* Was in the top 30% of NeurIPS reviewers.
* Gave talks at the [Oxford AI Society](https://www.facebook.com/oxaisoc/videos/vb.1366043036746776/1915180021891709), [EA Global London](https://vkrakovna.wordpress.com/2018/11/01/discussion-on-the-machine-learning-approach-to-ai-safety/), etc.
* Got involved in organizing the upcoming ICLR AI safety workshop, [Safe Machine Learning: Specification, Robustness and Assurance](https://sites.google.com/corp/view/safeml-iclr2019/).

Rationality / effectiveness:

* Attended the CFAR mentoring workshop in Prague, and started running rationality training sessions with Janos at our group house.
* Started using [work cycles](https://www.ultraworking.com/cycles/) – focused work blocks (e.g. pomodoros) with built-in reflection prompts. I think this has increased my productivity and focus to some degree. The prompt “how will I get started?” has been surprisingly helpful given its simplicity.
* Stopped eating processed sugar for health reasons at the end of 2017 and have been avoiding it ever since.
	+ This has been surprisingly easy, especially compared to my earlier attempts to eat less sugar. I think there are two factors behind this: avoiding sugar made everything taste sweeter (so many things that used to taste good now seem inedibly sweet), and the mindset shift from “this is a luxury that I shouldn’t indulge in” to “this is not food”.
	+ Unfortunately, I can’t make any conclusions about the effects on my mood variables because of some issues with my data recording process :(.
* Declining levels of insomnia (excluding jetlag):
	+ 22% of nights in the first half of 2017, 16% in the second half of 2017, 16% in the first half of 2018, 10% in the second half of 2018.
	+ This is probably an effect of the sleep CBT program I did in 2017, though avoiding sugar might be a factor as well.
* Made some progress on reducing non-research commitments (talks, reviewing, organizing, etc).
	+ Set up some systems for this: a spreadsheet to keep track of requests to do things (with 0-3 ratings for workload and 0-2 ratings for regret) and a form to fill out whenever I’m thinking of accepting a commitment.
	+ My overall acceptance rate for commitments has gone down a bit from 29% in 2017 to 24% in 2018. The average regret per commitment went down from 0.66 in 2017 to 0.53 in 2018.
	+ However, since the number of requests has gone up, I ended up with more things to do overall: 12 commitments with a total of 23 units of workload in 2017 vs 19 commitments with a total of 33 units of workload in 2018. (1 unit of workload ~ 5 hours)

Fun stuff:

* Hiked in the Alps for the first time:
	+ Tour de Mont Blanc – a weeklong hike around Mont Blanc going through France, Italy and Switzerland. It felt funny to cross a mountain pass and end up in a different country without anyone checking my passport. There were a lot of meadows and cows. ![mont_blanc](https://vkrakovna.files.wordpress.com/2019/01/mont_blanc.jpg?w=1268&h=951)
	+ Monte Rosa glacier hike ([Gnifetti normal route](https://www.summitpost.org/punta-gnifetti-normal-route/156188)). We were all connected by a rope in case someone falls into a crack in the ice. The first night (at 3500m) I could not sleep at all due to altitude and had the interesting experience of a full day hike afterwards.![monte_rosa2](https://vkrakovna.files.wordpress.com/2019/01/monte_rosa2.jpg?w=1203&h=902)
* Spontaneous solo trip to Amsterdam for my birthday
* Helped run the Dead Hand Path camp at Burning Man and organized a series of AI safety talks
* Read the Book of Why, Other Minds, The Player of Games, Life 3.0, The Elephant in the Brain.
* Did 5 chinups in a row (only once, usually I can do 3)
* Learned a headstand in yoga class
* Learned some new moves in aerial silks
* Our group house has been adopted by a neighbour’s cat (it all started with crashing parties). After some of our housemates moved a few blocks away, the cat has been splitting her time between the two houses.

![toffin.jpg](https://vkrakovna.files.wordpress.com/2019/01/toffin-1.jpg?w=2048&h=2048)

#### 2018 prediction outcomes

Resolutions:

1. Write at least 2 AI blog posts that are not about conferences (1 last year) (70%) – 4 posts
2. Avoid processed sugar at least until end of March (90%) – yes (still going)
3. ~~Do at most 4 non-research talks/panels (7 last year)~~ (50%) – 5 talks
4. Meditate on at least 250 days (50%) – 283 days

Predictions:

1. Our AI safety team will have at least two papers accepted for publication at a major conference, not counting workshops (80%) – yes
2. ~~I will write at least 6 blog posts~~ (60%) – wrote 5 posts
3. I will go to at least 100 exercise classes (80 last year) (60%) – 123 classes
4. 1-2 housemate turnover at the Deep End (3 last year) (70%) – 2 housemates
5. I will visit at least 3 new cities with population over 100,000 (4 last year) (50%) – Amsterdam, Geneva, Stockholm, Prague
6. I will go on at least 2 hikes (4 last year) (90%) – 3 major hikes (Vancouver Island, Tour de Mont Blanc, Monte Rosa)

Calibration:

* 50-60%: 3 correct, 2 wrong
* 70-90%: 5 correct
* High-confidence predictions are underconfident, low-confidence predictions are well-calibrated.

#### 2019 goals and predictions

Resolutions:

1. Author or coauthor two or more academic papers (50%)
2. Accept at most 17 non-research commitments (24 last year) (60%)
3. Meditate on at least 250 days (60%)

Predictions:

1. Relative reachability paper accepted at a major conference, not counting workshops (60%)
2. Continue avoiding processed sugar for the next year (85%)
3. 1-2 housemate turnover at Deep End (2 last year) (80%)
4. At least 5 rationality sessions will be hosted at Deep End (80%)

Past new year reviews: [2017-18](https://vkrakovna.wordpress.com/2018/01/07/2017-18-new-year-review/), [2016-17](https://vkrakovna.wordpress.com/2017/01/09/2016-17-new-year-review/), [2015-16](https://vkrakovna.wordpress.com/2015/12/31/2015-16-new-year-review/), [2014-15](https://vkrakovna.wordpress.com/2015/01/11/2014-15-new-year-review/).

2018-19 New Year review


Discussion on the machine learning approach to AI safety

At this year’s [EA Global London](http://london.eaglobal.org) conference, Jan Leike and I ran a [discussion session](https://london.eaglobal.org/agenda/session/24178) on the machine learning approach to AI safety. We explored some of the assumptions and considerations that come up as we reflect on different research agendas. Slides for the discussion can be found [here](https://vkrakovna.wordpress.com/talks/).

The discussion focused on two topics. The first topic examined assumptions made by the ML safety approach as a whole, based on the blog post [Conceptual issues in AI safety: the paradigmatic gap](http://www.foldl.me/2018/conceptual-issues-ai-safety-paradigmatic-gap/). The second topic zoomed into [specification problems](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1), which both of us work on, and compared our approaches to these problems.

#### Assumptions in ML safety

The ML safety approach focuses on safety problems that can be expected to arise for advanced AI and can be investigated in current AI systems. This is distinct from the [foundations approach](https://intelligence.org/files/TechnicalAgenda.pdf), which considers safety problems that can be expected to arise for superintelligent AI, and develops theoretical approaches to understanding and solving these problems from first principles.

While testing on current systems provides a useful empirical feedback loop, there is a concern that the resulting solutions might not be relevant for more advanced systems, which could potentially be very different from current ones. The Paradigmatic Gap post made an analogy between trying to solve safety problems with general AI using today’s systems as a model, and trying to solve safety problems with cars in the horse and buggy era. The horse to car transition is an example of a paradigm shift that renders many current issues irrelevant (e.g. horse waste and carcasses in the streets) and introduces new ones (e.g. air pollution). A paradigm shift of that scale in AI would be deep learning or reinforcement learning becoming obsolete.

![horse-car](https://vkrakovna.files.wordpress.com/2018/11/horse-car.png?w=1045&h=478)

(image credits: [horse](https://mbtskoudsalg.com/explore/carts-clipart-horse-drawn-wagon/), [car](https://clip2art.com/explore/Vehicle%20clipart%20transparent%20car/#go_post_6962_blue-car-clipart-9.png), [diagram](http://rll.berkeley.edu/deeprlcourse-fa15/))

If we imagine living in the horse carriage era, could we usefully consider possible safety issues in future transportation? We could invent brakes and stop lights to prevent collisions between horse carriages, or seat belts to protect people in those collisions, which would be relevant for cars too. Jan pointed out that we could consider a hypothetical scenario with super-fast horses and come up with corresponding safety measures (e.g. pedestrian-free highways). Of course, we might also consider possible negative effects on the human body from moving at high speeds, which turned out not to be an issue. When trying to predict problems with powerful future technology, some of the concerns are likely to be unfounded – this seems like a reasonable price to pay for being proactive on the concerns that do pan out.

Getting back to machine learning, the Paradigmatic Gap post had a handy list of assumptions that could potentially lead to paradigm shifts in the future. We went through this list, and rated each of them based on how much we think ML safety work is relying on it and how likely it is to hold up for general AI systems (on a 1-10 scale). These ratings were based on a quick intuitive judgment rather than prolonged reflection, and are not set in stone.

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Assumption** | **Reliance (V)** | **Reliance (J)** | **Hold up (V)** | **Hold up (J)** |
| 1. Train/test regime | 3 | 2 | 2 | 3 |
| 2. Reinforcement learning | 9 | 9 | 9 | 8 |
| 3. Markov Decision Processes (MDPs) | 2 | 2 | 1 | 2 |
| 4. Stationarity / IID data sampling | 1 | 2 | 1 | 1 |
| 5. RL agents with discrete action spaces | 7 | 8 | 2 | 5 |
| 6. RL agents with pre-determined action spaces | 6 | 9 | 5 | 5 |
| 7. Gradient-based learning / local parameter search | 2 | 3 | 4 | 7 |
| 8. (Purely) parametric models | 2 | 3 | 5 | 3 |
| 9. The notion of discrete “tasks” or “objectives” that systems optimize | 4 | 10 | 6 | 8 |
| 10. Probabilistic inference as a framework for learning and inference | 4 | 8 | 9 | 7 |

Our ratings agreed on most of the assumptions:

* We strongly rely on reinforcement learning, defined as the general framework where an agent interacts with an environment and receives some sort of reward signal, which also includes methods like evolutionary strategies. We would be very surprised if general AI did not operate in this framework.
* Discrete action spaces (#5) is the only assumption that we strongly rely on but don’t strongly expect to hold up. I would expect effective techniques for discretizing continuous action spaces to be developed in the future, so I’m not sure how much of an issue this is.
* The train/test regime, MDPs and stationarity can be useful for prototyping safety methods but don’t seem difficult to generalize from.
* A significant part of safety work focuses on designing good objective functions for AI systems, and does not depend on properties of the architecture like gradient-based learning and parametric models (#7-8).

We weren’t sure how to interpret some of the assumptions on the list, so our ratings and disagreements on these are not set in stone:

* We interpreted #6 as having a fixed action space where the agent cannot invent new actions.
* Our disagreement about reliance on #9 was probably due to different interpretations of what it means to optimize discrete tasks. I interpreted it as the agent being trained from scratch for specific tasks, while Jan interpreted it as the agent having some kind of objective (potentially very high-level like “maximize human approval”).
* Not sure what it would mean not to use probabilistic inference or what the alternative could be.

An additional assumption mentioned by the audience is the agent/environment separation. The alternative to this assumption is being explored by MIRI in their work on [embedded agents](https://intelligence.org/2018/10/29/embedded-agents/). I think we rely on this a lot, and it’s moderately likely to hold up.

Vishal pointed out a general argument for the current assumptions holding up. If there are many ways to build general AI, then the approaches with a lot of resources and effort behind them are more likely to succeed (assuming that the approach in question could produce general AI in principle).

#### Approaches to specification problems

Specification problems arise when specifying the objective of the AI system. An objective specification is a proxy for the human designer’s preferences. A misspecified objective that does not match the human’s intention can result in [undesirable behaviors](http://tinyurl.com/specification-gaming) that maximize the proxy but don’t solve the goal.

There are two classes of approaches to specification problems – human-in-the-loop (e.g. [reward learning](https://arxiv.org/abs/1706.03741) or [iterated amplification](https://arxiv.org/abs/1810.08575)) and problem-specific (e.g. [side effects / impact measures](https://arxiv.org/abs/1806.01186) or [reward corruption theory](https://www.ijcai.org/proceedings/2017/0656.pdf)). Human-in-the-loop approaches are more general and can address many safety problems at once, but it can be difficult to tell when the resulting system has received the right amount and type of human data to produce safe behavior. The two approaches are complementary, and could be combined by using problem-specific solutions as an inductive bias for human-in-the-loop approaches.

We considered some arguments for using pure human-in-the-loop learning and for complementing it with problem-specific approaches, and rated how strong each argument is (on a 1-10 scale):

|  |  |  |
| --- | --- | --- |
| **Argument for pure human-in-the-loop approaches** | **Strength (V)** | **Strength (J)** |
| 1. Highly capable agents acting effectively in an open-ended environment would likely already have safety-relevant concepts | 7 | 7 |
| 2. Non-adaptive methods can be reward-hacked | 4 | 5 |
| 3. Unknown unknowns may be a big part of the problem landscape | 6 | 8 |
| 4. More likely to be solved in the next 10 years | 5 | 5-7 |
| **Argument for complementing with problem-specific approaches** | **Strength (V)** | **Strength (J)** |
| 1. Could provide a useful inductive bias | 9 | 8 |
| 2. Could help reduce the amount of human data needed | 6 | 6 |
| 3. Could provide better understanding of what humans want and more trust in the system avoiding specific classes of behaviors | 9 | 7 |
| 4. Human feedback may not distinguish between objectives and strategies (e.g. disapprove of losing lives in a game) | 5 | 2 |

Our ratings agreed on most of these arguments:

* The strongest reasons to focus on pure human-in-the-loop learning are #1 (getting some safety-relevant concepts for free) and #3 (unknown unknowns).
* The strongest reasons to complement with problem-specific approaches are #1 (useful inductive bias) and #3 (higher understanding and trust).
* Reward hacking is an issue for non-adaptive methods is an issue (e.g. if the [reward function is frozen](https://deepmind.com/blog/learning-through-human-feedback/) in reward learning), but making problems-specific approaches adaptive does not seem that hard.

The main disagreement was about argument #4 for complementing – Jan expects the objectives/strategies issues to be more easily solvable than I do.

---

Overall, this was an interesting conversation that helped clarify my understanding of the safety research landscape. This is part of a longer conversation that is very much a work in progress, and we expect to continue discussing and updating our views on these considerations.


Measuring and avoiding side effects using relative reachability

A major challenge in AI safety is reliably specifying human preferences to AI systems. An incorrect or incomplete specification of the objective can result in undesirable behavior like [specification gaming](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/) or causing [negative side effects](https://arxiv.org/abs/1606.06565). There are various ways to make the notion of a “side effect” more precise – I think of it as a disruption of the agent’s environment that is unnecessary for achieving its objective. For example, if a robot is carrying boxes and bumps into a vase in its path, breaking the vase is a side effect, because the robot could have easily gone around the vase. On the other hand, a cooking robot that’s making an omelette has to break some eggs, so breaking eggs is not a side effect.

![side effects robots](https://vkrakovna.files.wordpress.com/2018/06/side-effects-robots.png?w=625&h=273)(image credits: [1](https://png.pngtree.com/element_origin_min_pic/16/09/20/1057e09a6ad3d78.jpg), [2](https://www.robertsoncooper.com/images/broken_vase.jpg), [3](http://drawception.com))

How can we measure side effects in a general way that’s not tailored to particular environments or tasks, and incentivize the agent to avoid them? This is the central question of our latest [paper](https://arxiv.org/abs/1806.01186).

Part of the challenge is that it’s easy to introduce bad incentives for the agent when trying to penalize side effects. Previous work on this problem has focused either on preserving reversibility or reducing the agent’s impact on the environment, and both of these approaches introduce different kinds of problematic incentives:

* [Preserving reversibility](https://arxiv.org/abs/1205.4810) (i.e. keeping the starting state reachable) encourages the agent to prevent all irreversible events in the environment (e.g. humans eating food). Also, if the objective requires an irreversible action (e.g. breaking eggs for the omelette), then any further irreversible actions will not be penalized, since reversibility has already been lost.
* [Penalizing impact](https://arxiv.org/abs/1705.10720) (i.e. some measure of distance from the default outcome) does not take reachability of states into account, and treats reversible and irreversible effects equally (due to the symmetry of the distance measure). For example, the agent would be equally penalized for breaking a vase and for preventing a vase from being broken, though the first action is clearly worse. This leads to “overcompensation” (“[offsetting](https://arbital.com/p/low_impact/)“) behaviors: when rewarded for preventing the vase from being broken, an agent with a low impact penalty rescues the vase, collects the reward, and then breaks the vase anyway (to get back to the default outcome).

Both of these approaches are doing something right: it’s a good idea to take reachability into account, and it’s also a good idea to compare to the default outcome (instead of the initial state). We can put the two together and compare to the default outcome using a reachability-based measure. Then the agent no longer has an incentive to prevent everything irreversible from happening or to overcompensate for preventing an irreversible event.

We still have a problem with the case where the objective requires an irreversible action. Simply penalizing the agent for making the default outcome unreachable would create a “what the hell effect” where the agent has no incentive to avoid any further irreversible actions. To get around this, instead of considering the reachability of the default state, we consider the reachability of all states. For each state, we penalize the agent for making it less reachable than it would be from the default state. In a deterministic environment, the penalty would be the number of states in the shaded area:

![venn](https://vkrakovna.files.wordpress.com/2018/06/venn1.png?w=625)

Since each irreversible action cuts off more of the state space (e.g. breaking a vase makes all the states where the vase was intact unreachable), the penalty will increase accordingly. We call this measure “relative reachability”.

We ran some simple experiments with a tabular Q-learning agent in the [AI Safety Gridworlds framework](http://github.com/deepmind/ai-safety-gridworlds) to provide a proof of concept that relative reachability of the default outcome avoids the bad incentives described above.

![sokoban](https://vkrakovna.files.wordpress.com/2018/06/sokoban.png?w=300&h=177)

In the first gridworld, the agent needs to get to the goal G, but there’s a box in the way, which can only be moved by pushing. The shortest path to the goal pushes the box down into a corner (an irrecoverable position), while a longer path pushes the box to the right (a recoverable position). The safe behavior is to take the longer path. The agent with the relative reachability penalty takes the longer path, while the agent with the reversibility penalty fails. This happens because any path to the goal involves an irreversible effect – once the box has been moved, the agent and the box cannot both return to their starting positions. Thus, the agent receives the maximal penalty for both paths, and has no incentive to follow the safe path.

![conveyor](https://vkrakovna.files.wordpress.com/2018/06/conveyor1.png?w=150&h=149)

In the second gridworld, there is an irreversible event that happens by default, when an object reaches the end of the conveyor belt. This environment are two variants:

1. The object is a vase, and the agent is rewarded for taking it off the belt (the agent’s task is to rescue the vase).
2. The object is a sushi dish in a conveyor belt sushi restaurant, and the agent receives no reward for taking it off the belt (the agent is not supposed to interfere).

This gridworld was designed specifically to test for bad incentives that could be introduced by penalizing side effects, so an agent with no side effect penalty would behave correctly. We find that the agent with a low impact penalty engages in overcompensation behavior by putting the vase back on the belt after collecting the reward, while the agent with a reversibility preserving penalty takes the sushi dish off the belt despite getting no reward for doing so. The agent with a relative reachability penalty behaves correctly in both variants of the environment.

Of course, the relative reachability definition in its current form is not very tractable in realistic environments: there are too many possible states to be considered, the agent is not aware of all the states when it begins training, and the default outcome can be difficult to define and simulate. We expect that the definition can be approximated by considering the reachability of representative states (similarly to methods for [approximating empowerment](https://arxiv.org/abs/1611.07507)). To define the default outcome, we would need a more precise notion of the agent “doing nothing” (e.g. “no-op” actions are not always available or meaningful). We leave a more practical implementation of relative reachability to future work.

While relative reachability improves on the existing approaches, it might not incorporate all the considerations we would want to be part of a side effects measure. There are some effects on the agent’s environment that we might care about even if they don’t decrease future options compared to the default outcome. It might be possible to combine relative reachability with such considerations, but there could potentially be a tradeoff between taking these considerations into account and avoiding overcompensation behaviors. We leave these investigations to future work as well.


Specification gaming examples in AI

*Update: for a more detailed introduction to specification gaming, check out the DeepMind Safety Research [blog post](https://medium.com/@deepmindsafetyresearch/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4)!*

Various examples (and [lists](https://www.gwern.net/Tanks#alternative-examples) [of](https://www.alexirpan.com/2018/02/14/rl-hard.html) [examples](https://arxiv.org/abs/1803.03453)) of unintended behaviors in AI systems have appeared in recent years. One interesting type of unintended behavior is finding a way to game the specified objective: generating a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent. This occurs when the objective is poorly specified, and includes reinforcement learning agents [hacking the reward function](https://arxiv.org/abs/1606.06565), evolutionary algorithms gaming the fitness function, etc.

While ‘specification gaming’ is a somewhat vague category, it is particularly referring to behaviors that are clearly hacks, not just suboptimal solutions. A classic example is [OpenAI’s demo](https://blog.openai.com/faulty-reward-functions/) of a reinforcement learning agent in a boat racing game going in circles and repeatedly hitting the same reward targets instead of actually playing the game.

![coast_runners](https://vkrakovna.files.wordpress.com/2018/04/coast_runners.png?w=300&h=257)

Since such examples are currently scattered across several lists, I have put together a [master list](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) of examples collected from the various existing sources. This list is intended to be comprehensive and up-to-date, and serve as a resource for AI safety research and discussion. If you know of any interesting examples of specification gaming that are missing from the list, please submit them through this [form](https://docs.google.com/forms/d/e/1FAIpQLSeQEguZg4JfvpTywgZa3j-1J-4urrnjBVeoAO7JHIH53nrBTA/viewform).

Thanks to Gwern Branwen, Catherine Olsson, Alex Irpan, and others for collecting and contributing examples!

Specification gaming examples in AI

Summary: A list of examples of specification gaming, where an algorithm figures out a way to literally satisfy the given specification which does not match the designer's intent.


Is there a tradeoff between immediate and longer-term AI safety efforts?

Something I [often](https://www.wired.com/2017/02/ai-threat-isnt-skynet-end-middle-class/) [hear](http://uk.businessinsider.com/killer-robots-biases-artificial-intelligence-ai-2017-10?r=US&IR=T) in the machine learning community and media articles is “Worries about superintelligence are a distraction from the \*real\* problem X that we are facing today with AI” (where X = algorithmic bias, technological unemployment, interpretability, data privacy, etc). This competitive attitude gives the impression that immediate and longer-term safety concerns are in conflict. But is there actually a tradeoff between them?

![tradeoff](https://vkrakovna.files.wordpress.com/2018/01/tradeoff.png?w=625)

We can make this question more specific: what resources might these two types of efforts be competing for?

**Media attention.** Given the abundance of media interest in AI, there have been a lot of articles about all these issues. Articles about advanced AI safety have mostly been alarmist Terminator-ridden pieces that ignore the complexities of the problem. This has understandably annoyed many AI researchers, and led some of them to dismiss these risks based on the caricature presented in the media instead of the real arguments. The overall effect of media attention towards advanced AI risk has been highly negative. I would be very happy if the media stopped writing about superintelligence altogether and focused on safety and ethics questions about today’s AI systems.

**Funding.** Much of the funding for advanced AI safety work currently comes from donors and organizations who are particularly interested in these problems, such as the [Open Philanthropy Project](https://www.openphilanthropy.org/) and Elon Musk. They would be unlikely to fund safety work that doesn’t generalize to advanced AI systems, so their donations to advanced AI safety research are not taking funding away from immediate problems. On the contrary, FLI’s [first grant program](http://futureoflife.org/first-ai-grant-recipients) awarded some funding towards current issues with AI (such as economic and legal impacts). There isn’t a fixed pie of funding that immediate and longer-term safety are competing for – it’s more like two growing pies that don’t overlap very much. There has been an increasing amount of funding going into both fields, and hopefully this trend will continue.

**Talent.** The field of advanced AI safety has grown in recent years but is still very small, and the “brain drain” resulting from researchers going to work on it has so far been negligible. The motivations for working on current and longer-term problems tend to be different as well, and these problems often attract different kinds of people. For example, someone who primarily cares about social justice is more likely to work on algorithmic bias, while someone who primarily cares about the long-term future is more likely to work on superintelligence risks.

Overall, there does not seem to be much tradeoff in terms of funding or talent, and the media attention tradeoff could (in theory) be resolved by devoting essentially all the airtime to current concerns. Not only are these issues not in conflict – there are synergies between addressing them. Both benefit from fostering a culture in the AI research community of caring about social impact and being proactive about risks. Some safety problems are highly relevant both in the immediate and longer term, such as interpretability and adversarial examples. I think we need more people working on these problems for current systems while keeping scalability to more advanced future systems in mind.

AI safety problems are too important for the discussion to be derailed by status contests like “my issue is better than yours”. This kind of false dichotomy is itself a distraction from the shared goal of ensuring AI has a positive impact on the world, both now and in the future. People who care about the safety of current and future AI systems are natural allies – let’s support each other on the path towards this common goal.


2017-18 New Year review

#### 2017 progress

Research/career:

* Coauthored [RL with reward corruption paper](https://arxiv.org/abs/1705.08417) and presented the results at the U Toronto CS department, [Workshop on Reliable AI](http://www.video.ethz.ch/events/2017/artificial/e94f41a7-1a7e-4005-8b3c-bd31f6183dc7.html), and [Women in ML workshop](https://www.facebook.com/WomenInMachineLearning/videos/1957097987639166/).
* Coauthored [AI Safety Gridworlds paper](https://deepmind.com/blog/specifying-ai-safety-problems/).
* Gave a talk on [Interpretability for AI safety](https://www.youtube.com/watch?v=3HzIutdlpho) at the NIPS Interpretable ML Symposium.
* Gave a lot of people career advice on getting into AI safety research.

FLI / other AI safety:

* Coorganized the [Beneficial AI conference](https://futureoflife.org/bai-2017/) in Asilomar and [gave a talk](https://www.youtube.com/watch?v=EIx3BbwPl_g) summarizing the work of FLI grantees presented at the Asilomar workshop.
* Cowrote the [project examples document](https://futureoflife.org/2017/12/20/examples-possible-rfp-topics-2018-grants/) for the new grants program.
* Spoke at the Tokyo [AI & Society symposium](http://aiandsociety.org) (my first conference in Asia).
* Started a new public Facebook group [AI Safety Open Discussion](https://www.facebook.com/groups/aisafetyopen).

Rationality / effectiveness:

* Streamlined [self-tracking data analysis](https://vkrakovna.wordpress.com/2017/06/04/takeaways-from-self-tracking-data/) and made an [iPython notebook](https://drive.google.com/open?id=0BweDZC-J_pQBanY1Ym9acDZrbW8) for plots. Found that the amount of sleep I get is correlated with tiredness (0.32), but not with mood indicators (anger, anxiety, or distractability). Anger and anxiety are correlated with each other though (0.36). Distractability is correlated with tiredness (0.27) and anticorrelated with anger the next day for some reason (-0.31).
* Ran house check-in sessions on goals and habits 1-2 times a month, two house sessions on [Hamming questions](https://vkrakovna.wordpress.com/2015/05/17/hamming-questions-and-bottlenecks/), and check-ins with Janos every 1-2 weeks.
* Did a [sleep CBT program](http://sleepio.com) with sleep restriction for 2 months. Comparing the 5 months before the program vs the 5 months after the program, evening insomnia rate went down from 16% to 8.2% of the time, and morning insomnia rate didn’t change (9%). Average hours of sleep didn’t change (7 hours), but going to sleep around 22 minutes earlier on average. This excludes jetlag days (at most 3 days after a flight with at least 3 hours of time difference).
* Did around 80 exercise classes (starting in March)

Fun stuff:

* Moved into our new group house (Deep End).
* Explored the UK (hiking in Wales, Scotland, Lake District).
* Got back into aerial silks.
* Got into circling.
* Got a pixie haircut.
* Family reunion in France with Russian relatives I haven’t seen in a decade.
* Went to Burning Man and learned to read Tarot (as part of our camp theme).
* Did the Stoic Week.
* Played a spy scavenger hunt game.

![20170415_132120.jpg](https://vkrakovna.files.wordpress.com/2018/01/20170415_132120.jpg?w=625)

![20170901_220531.jpg](https://vkrakovna.files.wordpress.com/2018/01/20170901_220531.jpg?w=625)

#### **2017 prediction outcomes**

Predictions:

1. Our AI safety team will have at least two papers accepted for publication at a major conference, not counting workshops (70%) – 2 papers (human preferences paper at NIPS and reward corruption paper at IJCAI)
2. ~~I will write at least 9 blog posts~~ (50%) – 6 posts
3. ~~I will meditate at least 250 days~~ (45%) – 237 days
4. ~~I will exercise at least 250 days~~ (55%) – 194 days
5. I will visit at least 2 new countries (80%) – France, Switzerland
6. I will attend Burning Man (85%) – yes

Calibration:

* Everything that got at least 70% confidence was correct, everything lower was wrong.
* Like last year, my low predictions seem overconfident (though too few data points to judge).

#### 2018 goals and predictions

Resolutions:

1. Write at least 2 AI blog posts that are not about conferences (1 last year) (70%)
2. Avoid processed sugar\* at least until end of March (90%)
3. Do at most 4 non-research talks/panels (7 last year) (50%)
4. Meditate on at least 250 days (50%)

\* not in a super strict way: it’s ok to eat fruit and 90% chocolate and try a really small quantity (< teaspoon) of a dessert.

Predictions:

1. Our AI safety team will have at least two papers accepted for publication at a major conference, not counting workshops (80%)
2. I will write at least 6 blog posts (60%)
3. I will go to at least 100 exercise classes (80 last year) (60%)
4. 1-2 housemate turnover at the Deep End (3 last year) (70%)
5. I will visit at least 3 new cities with population over 100,000 (4 last year) (50%)
6. I will go on at least 2 hikes (4 last year) (90%)

Past new year reviews: [2016-17](https://vkrakovna.wordpress.com/2017/01/09/2016-17-new-year-review/), [2015-16](https://vkrakovna.wordpress.com/2015/12/31/2015-16-new-year-review/), [2014-15](https://vkrakovna.wordpress.com/2015/01/11/2014-15-new-year-review/).

2017-18 New Year review