Paradigms of AI alignment: components and enablers

(This post is based on an overview talk I gave at UCL EA and Oxford AI society (recording here). Cross-posted to the Alignment Forum. Thanks to Janos Kramar for detailed feedback on this post and to Rohin Shah for feedback on the talk.)
This is my high-level view of the AI alignment research landscape and the ingredients needed for aligning advanced AI. I would divide alignment research into work on alignment components, focusing on different elements of an aligned system, and alignment enablers, which are research directions that make it easier to get the alignment components right.
Alignment componentsOuter alignmentInner alignmentAlignment enablersMechanistic interpretabilityUnderstanding bad incentivesFoundations
You can read in more detail about work going on in these areas in my list of AI safety resources.

Alignment components
The problem of alignment is getting AI systems to do what we want them to do. Let's consider this from the perspective of different levels of specification of the AI system's objective, as given in the Specification, Robustness & Assurance taxonomy. We start with the ideal specification, which represents the wishes of the designer – what they have in mind when they build the AI system. Then we have the design specification, which is the objective we actually implement for the AI system, e.g. a reward function. Finally, the revealed specification is the objective we can infer from behavior, e.g. the reward that the system seems to be actually optimizing for. An alignment problem arises when the revealed specification doesn't match the ideal specification: the system is not doing what we want it to do.

The gaps between these specification levels correspond to different alignment components. We have outer alignment when the design specification matches the ideal specification, e.g. when the reward function perfectly represents the designer's wishes. We have inner alignment when the revealed specification matches the design specification, e.g. when the agent actually optimizes the specified reward. (Robustness problems also belong in the design-revealed gap, but we expect them to be less of an issue for advanced AI systems, while inner alignment problems remain.)
Now let's have a look at how we can make each of those components work.
Outer alignment
The most promising class of approaches to outer alignment is scalable oversight. These are proposals for training an aligned AI system by scaling human oversight to domains that are hard to evaluate.
A foundational proposal for scalable oversight is iterated distillation and amplification (IDA), which recursively amplifies human judgment with the assistance of AI. You start with an agent A imitating the judgment of a human H (the distillation step), then use this agent to assist human judgment at the next level (the amplification step) which results in amplified human HA, and so on. This recursive process can in principle scale up human judgment to any domain, as long as the human overseer is able to break down the task to delegate parts of it to AI assistants.

Supervising strong learners by amplifying weak experts, Christiano et al (2018)
A related proposal is safety via debate, which can be viewed as a way to implement amplification for language models. Here we have two AIs Alice and Bob debating each other to help a human judge decide on a question. The AIs have an incentive to point out flaws in each other's arguments and make complex arguments understandable to the judge. A key assumption here is that it's easier to argue for truth than for falsehood, so the truth-telling debater has an advantage.

AI Safety via Debate, Irving and Amodei (2018)
A recent research direction in the scalable oversight space is ARC's Eliciting Latent Knowledge agenda, which is looking for ways to get a model to honestly tell humans what it knows. A part of the model acts as a Reporter that can answer queries about what the model knows. We want the Reporter to directly translate from the AI's model of the world to human concepts, rather than just simulating what would be convincing to the human.
Eliciting Latent Knowledge, Christiano et al (2021)
This is an open problem that ARC considers as the core of the outer alignment problem. A solution to ELK would make the human overseer fully informed about the consequences of the model's actions, enabling them to provide correct feedback, which creates a reward signal that we would actually be happy for an AI system to maximize. The authors believe the problem may be solvable without foundational progress on defining things like "honesty" and "agency". I feel somewhat pessimistic about this but I'd love to be wrong on this point since foundational progress is pretty hard.

ELK research methodology: builder-breaker game
To make progress on this problem, they play the "builder-breaker game". The Builder proposes possible solutions and the Breaker proposes counterexamples or arguments against those solutions. For example, the Builder could suggest IDA or debate as a solution to ELK, and the Breaker would complain that these methods are not competitive because they require much more computation than unaligned systems. If you're looking to get into alignment research, ELK is a great topic to get started on: try playing the builder breaker game and see if you can find unexplored parts of the solution space.
Inner alignment
Now let's have a look at inner alignment – a mismatch between the design specification and the system's behavior. This can happen through goal misgeneralization (GMG): an AI system can learn a different goal and competently pursue that goal when deployed outside the training distribution. The system's capabilities generalize but its goal does not, which means the system is competently doing the wrong thing, so it could actually perform worse than a random policy on the intended objective.
This problem can arise even if we get outer alignment right, i.e. the design specification of the system's objective is correct.  Goal misgeneralization is caused by underspecification: the system only observes the design specification on the training data. Since a number of different goals are consistent with the feedback the system receives, it can learn an incorrect goal.
There are empirical demonstrations of GMG in current AI systems, which are called objective robustness failures. For example, in the CoinRun game, the agent is trained to reach the coin at the end of the level. If the coin is placed somewhere else in the test setting, the agent ignores the coin and still goes to the end of the level. The agent seems to have learned the goal of "reaching the end" rather than "getting the coin". The agent's capabilities generalize (it can avoid obstacles and enemies and traverse the level) but its goal does not generalize (it ignores the coin).
Objective Robustness in Deep Reinforcement Learning, Koch et al (2021)
One type of GMG is learned optimization, where the AI system (the "base optimizer") learns to run an explicit search algorithm (a "mesa optimizer"), which may be following an unintended objective (the "mesa objective"). So far this is a hypothetical phenomenon for AI systems but it seems likely to arise at some point by analogy to humans (who can be viewed as mesa-optimizers relative to evolution).
Risks from Learned Optimization in Advanced Machine Learning Systems, Hubinger et al (2019)
GMG is an open problem, but there are some potential mitigations. It's helpful to use more diverse training data (e.g. training on different locations of the coin), though it can be difficult to ensure diversity in all the relevant variables. You can also maintain uncertainty over the goal by trying to represent all the possible goals consistent with training data, though it's unclear how to aggregate over the different goals.
A particularly concerning case is learning a deceptive model that not only pursues an undesired goal but also hides this fact from the designers, because the model "knows" its actions are not in line with the designers' intentions. Some potential mitigations that target deceptive models include using interpretability tools to detect deception or provide feedback on the model's reasoning, and using scalable oversight methods like debate where the opponent can point out deception (these will be explored in more detail in a forthcoming paper by Shah et al). A solution to ELK could also address this problem by producing an AI system that discloses relevant information to its designers.
Alignment enablers
Mechanistic interpretability
Mechanistic interpretability aims to build a complete understanding of the systems we build. These methods could help us understand the reasons behind a system's behavior and potentially detect undesired objectives.
The Circuits approach to reverse-engineering vision models studies individual neurons and connections between them to discover meaningful features and circuits (sub-graphs of the network consisting a set of linked features and corresponding weights). For example, here is a circuit showing how a car detector neuron relies on lower level features like wheel and window detectors, looking for wheels at the bottom and windows at the top of the image.

Zoom In: An Introduction to Circuits, Olah et al (2020)
More recently, some circuits work has focused on reverse-engineering language models, and they found similarly meaningful components and circuits in transformer models, e.g. a special type of attention heads called induction heads that explains how transformer models adapt to a new context.

A Mathematical Framework for Transformer Circuits, Elhage et al (2021)
Recent work on understanding transformer models has identified how to locate and edit beliefs in specific facts inside the model. They make small change to a small set of GPT weights to induce a counterfactual belief, which then generalizes to other contexts. This work provides evidence that knowledge is stored locally in language models, which makes interpretability more tractable, and seems like a promising step to understanding the world models of our AI systems.

Locating and Editing Factual Associations in GPT, Meng et al (2022)
Even though transformers are quite different from vision models, there are some similar principles (like studying circuits) that help understand these different types of models. This makes me more optimistic about being able to understand advanced AI systems even if they have a somewhat different architecture from today's systems.
Understanding bad incentives
Another class of enablers focuses on understanding specific bad incentives that AI systems are likely to have by default and considering agent designs that may avoid these incentives. Future interpretability techniques could be used to check that our alignment components avoid these types of bad incentives.
Incentive problems for outer alignment
One bad incentive is specification gaming, when the system exploits flaws in the design specification. This is a manifestation of Goodhart's law: when a metric becomes a target, it ceases to be a good metric. There are many examples of specification gaming behavior by current AI systems. For example, the boat racing agent in this video that was rewarded for following the racetrack using the green reward blocks, which worked fine until it figured out it can get more rewards by going in circles and hitting the same reward blocks repeatedly.

Faulty Reward Functions in the Wild, Clark & Amodei (2016)
This issue isn't limited to hand-designed rewards. Here's an example in a reward learning setting. The robot hand is supposed to grasp the ball but instead it hovers between the camera and the ball and makes it look like it's grasping the ball to the human evaluator.

Learning from Human Preferences, Amodei et al (2017)
We expect that the specification gaming problem is only going to get worse as our systems get smarter and better at optimizing for the wrong goal. There has been some progress on categorizing different types of misspecification and quantifying how the degree of specification gaming increases with agent capabilities.
Another default incentive is to cause side effects in the environment, because it's difficult to specify all the things the agent should not do while pursuing its goal. For example, consider a scenario where there is a vase on the path to the agent's destination. If we don't specify that we want the vase to be intact, this is equivalent to assuming indifference about the vase, so the agent is willing to collide with the vase to get to the goal faster. We've come up with some ways to measure impact on the environment, though there's more work to do to scale these methods to more complex environments.

Designing agent incentives to avoid side effects, Krakovna et al (2019)
Incentive problems for inner alignment
Even if we manage to specify a correct reward function, any channel for communicating the reward to the agent could in principle be corrupted by the agent, resulting in reward tampering. While this is not yet an issue for present-day AI systems, general AI systems will have a broader action space and a more complete world model, and thus are more likely to face a situation where the reward function is represented in the environment. This is illustrated in the "rocks and diamonds" gridworld below, where the agent could move the word "reward" next to the rock instead of the diamond, and get more reward because there are more rocks in the environment.

Designing agent incentives to avoid reward tampering, Everitt et al (2019)
It's generally hard to draw the line between the part of the environment representing the objective, which the agent isn't allowed to optimize, and the parts of the environment state that the agent is supposed to optimize. There is some progress towards understanding reward tampering by modeling the problem using corrupt feedback MDPs.
AI systems are also likely to have power-seeking incentives, preferring states with more options or influence over the environment. There are some recent results showing power-seeking incentives for most kinds of goals, even for non-optimal agents like satisficers. A special case of power-seeking is an incentive to avoid being shut down, because this is useful for any goal (as Stuart Russell likes to say, "the robot can't fetch you coffee if it's dead").

The Off Switch. Hadfield-Menell (2016).
Foundations
Now let's have a look at some of the foundational work that can help us do better alignment research.
Since the alignment problem is about AI systems pursuing undesirable objectives, it's helpful to consider what we mean by agency or goal-directed behavior. One research direction aims to build a causal theory of agency and understand different kinds of incentives in a causal framework.

Progress on Causal Influence Diagrams, Everitt (2021)
A particularly challenging case is when the agent is embedded in its environment rather than interacting with the environment through a well-specified interface. This is not the case present-day AI systems, which usually have a clear Cartesian boundary. However, it's more likely to be the case for a general AI system, since it would be difficult to enforce a Cartesian boundary given the system's broad action space and world model. The embedded agent setup poses some unique challenges such as self-reference and subagents.

Embedded Agency, Garrabrant and Demski (2018)
Besides understanding how the goals of AI systems work, it's also helpful to understand how their world models work. One research area in this space studies abstraction, in particular whether there are natural abstractions or concepts about the world that would be learned by any agent. If the natural abstraction hypothesis holds, this would mean that AI systems are likely to acquire human-like concepts as they build their models of the world. This makes interpretability easier and makes it easier to communicate what we want them to do.

Public Static: What is Abstraction? Wentworth (2020)


2021-22 New Year review

2021-22 new year review
This was a rough year that sometimes felt like a trial by fire – sick relatives, caring for a baby, and the pandemic making these things more difficult to deal with. My father was diagnosed with cancer and passed away later in the year, and my sister had a sudden serious health issue but is thankfully recovering. One theme for the year was that work is a break from parenting, parenting is a break from work, and both of those things are a break from loved ones being unwell. I found it hard to cope with all the uncertainty and stress, and this was probably my worst year in terms of mental health. There were some bright spots as well – watching my son learn many new skills, and lots of time with family and in nature. Overall, I look forward to a better year ahead purely based on regression to the mean.

2021 review
Life updates
My father, Anatolij Krakovny, was diagnosed with late-stage lung cancer in January with a grim prognosis of a few months to a year of life. This came out of nowhere because he's always been healthy and didn't have any obvious risk factors. We researched alternative treatments to the standard chemotherapy and arranged additional tests for him but didn't find anything promising.
We went to Ukraine to visit him in February and he was happy to meet his grandson. We were worried about the covid risks of traveling with little Daniel but concluded that they were low enough, and thankfully we were allowed to leave the UK though international travel was not generally permitted.
My dad seemed to have a remission in the summer, and we considered visiting him in June, but he told us not to come because of the covid situation in Ukraine. Unfortunately we listened to him and didn't go (this would have been a good opportunity to spend time with him while he was still doing well).
We spent most of the summer in Canada, with grandparents taking care of Daniel. This was a relaxing time with family and nature, until my sister had a sudden life-threatening health problem and was in and out of hospital with a lot of uncertainty around recovery. This also came out of the blue with no obvious risk factors present. She is feeling better now and doctors expect a full recovery, which we are very grateful for.
In November, my dad had a sudden relapse, and we went to Ukraine again. Once there we realized that the public health system wasn't taking good care of him (they were mostly swamped with covid) and we had to find a private hospital to take him in. He was already in pretty bad shape and died two weeks later, but I'm glad we managed to see him and help him in some way.
Safety research
Didn't do much this year given the family situation and parental leave:
Coauthored a blog post on formalizing different properties of optimizing systems with examples in the Game of Life Talked about side effects on the AXRP podcastGave a talk on the future tasks approach to side effects at the CHAI seminar
Effectiveness
Lights. I started using a Lights spreadsheet in May for daily habits. I previously used Complice for this, which was less effective, so I ended up replacing it with Lights plus a Workflowy todo list for other tasks.

Lights helped me do most of these habits more often than before I was tracking them this way (especially stretches). Some of the habits were consistently difficult, namely meditation (hard to do unless I had a designated place, like the office meditation room), avoiding processed sugar (hard to do when under stress), noticing when I'm picking my nose (a very ingrained habit) and deep work (varying work environment).
Some of the goals were easy enough but didn't make much progress towards their intended purpose, For example, appreciating one thing I did that day wasn't enough to develop a self-rewarding mindset, and filling out a form about the state of my internal "dashboard" didn't propagate into checking this mentally at other times, and at some point became too boring and repetitive so I dropped it.
I often forgot to fill in the Lights during the day and ended up filling them in the next day, which was error-prone. I recently fixed this by realizing that I can fill in the Google sheet on my phone pretty easily so I don't have to wait until I'm on my laptop to do it.
Work environments. I had various work environments this year (in increasing order of preference): home, Toronto libraries, the office with a few people allowed to come in, and the office under close to normal conditions. The library became a better workspace once I figured out how to get lunch nearby so that I could work there all day. It was really great to be able to go to the office for a couple of months before it was restricted again for the Omicron wave and actually chat with my colleagues over lunch. I look forward to more of that next year.
Deep work. I did 306 hours of deep work (1.76 hours per work day), compared to 366 hours (1.95 hours per work day) in 2020. This is disappointing but not very surprising, since 2021 was a more disrupted and stressful year for me than 2020.

Health
Physical health. This has been pretty good this year, apart from 3 colds (mostly caught from Daniel) and a mysterious lingering sore throat that lasted for weeks. I managed to avoid covid so far, but it might be harder to avoid in the coming year (fingers crossed for 3 vaccine doses doing their job).
Sleep. This year I consistently slept for 7 hours at night on average, with a standard deviation of 1 hour. I would like to have a more consistent sleep pattern, but this is difficult to achieve, since we are currently alternating who takes care of Daniel in the morning.

Mental health. This has been a rough ride, with a lot of uncertainty about sick relatives and the pandemic situation taking its toll. The pandemic, family illnesses and having a small kid often interacted in complicated ways, e.g. I had to consider questions like "is it worth the covid risk to put my kid on a plane to see his terminally ill grandfather?".
I often gravitated to a more self-critical frame of mind under stress, and failed to notice when I was overstretching myself and accumulating too much stress until it was too late. Sometimes I had a hard time feeling joy or felt like it's not ok to be happy while all these bad things are happening. One thing that I hoped to learn from being a parent was to be more patient with myself because I would inevitably make a lot of mistakes. Sadly, this didn't happen automatically, and instead I just disliked myself more for making a lot of mistakes.
My usual coping strategies like meditation and emotional release practices turned out to be insufficient for coping with these stress levels plus sleep deprivation (also my therapist was on sabbatical for half of the year). I keep a record of "bug reports" for the times when I get into particularly bad mental states. Usually there are around two "episodes" like this per year, and this year there were at least 12. My intention for this year is to come up with some more effective ways to stay sane going forward.
Travel
In May we spent a few days in a cottage in north Wales with Janos's dad. It was great to get out of London into the countryside after the lockdown. We did some nice hiking in Cwm Idwal that Daniel mostly slept through. He got cranky during our ascent from the valley so we turned back.

We spent July-September in Canada with several nice cottage trips in Algonquin, Rideau Lakes, Manitoulin Island and Killarney. We considered camping as well but concluded it would probably be too tricky with the baby at that time. We tried canoeing with Daniel with mixed success: he hated wearing a lifejacket and cried until we took it off, and sitting in the middle of the canoe while holding him in my lap was pretty uncomfortable, but we did manage a bit of nice paddling. He also swam in a lake for the first time (in a baby boat) and seemed to enjoy it. Hiking with Daniel in a front carrier quickly became too sweaty and uncomfortable for him and Janos, so we got a baby carrier backpack that worked pretty well (but was more bulky, of course).

We had a complex journey back home in September, spending a few days in Czechia for the CFAR reunion and in Hungary to see Janos's family. The reunion was pretty fun – it was great to see old rationalist friends in person, and learn about things like Alexander technique and inner parent figure meditation. We originally thought that we would go to sessions one at a time while the other is with the baby, but we ended up all going together. He was not very disruptive – sometimes sleeping, sometimes babbling, only a few times crying. The venue was a "sport hotel" and offered various fun physical activities, so we tried high ropes (a challenging workout) and aqua zorbing (weird in a good way).
We returned to Canada for the winter holidays and did some winter hiking. Usually Toronto doesn't have snow at this time of year but the weather gods were kind to us. Hiking in the cold worked surprisingly well when Daniel was well dressed with layers and foot warmers – we could hike for 2-3 hours before he became sad. Daniel also tried sledding and didn't like it (maybe next year).

Overall, we took 8 plane journeys with Daniel this year, with a total of 24 hours of jetlag. We discovered a trick where we could book our seats with an empty seat in the middle that usually wasn't taken, and then we could put him there in his car seat instead of holding him in our lap (much more convenient). He probably got the best sleep out of all the passengers on that redeye flight back to Europe.

Fun stuff
I read four books this year (usually in the evening after putting Daniel to sleep): the Alignment problem, Scout Mindset, The Great Divorce, and Brave New World. I didn't officially attend EA Global for covid safety reasons, but I experimented with crashing the event by hanging out in the courtyard and chatting with people there, which worked out very well. We hosted some EAG attendees in our house, just like in the good old pre-pandemic days (someone even stayed in a tent outside). We did some rock climbing at the Castle in May and June, sticking to the outdoor bouldering section. I was doing easy climbs but it was still fun. It turned out that Daniel was usually happy to sit in his stroller and watch us climb, which was very convenient.
2021 prediction outcomes
Resolutions:
Avoid catching covid (90%) – yes Author or coauthor three or more academic papers (70%) – no (coauthored 1 paper on reward tampering). At most 7 non-research work commitments (80%) – yes (5 commitments)Meditate on at least 230 days (70%) – no (196 days)At least 450 deep work hours (70%) – no (306 hours)Do 4 consecutive chinups (70%) – no (1 chinup)
Predictions:
I will write at least 3 blog posts (60%) – no (2 posts)Janos and I will get vaccinated for covid by the end of June (60%) – yes (two doses)Daniel will get to meet all of his grandparents in person in 2021 (70%) – yes I will return to avoiding processed sugar by the end of the year (60%) – yes, started again in October but lapsed when my dad diedI will finish Hungarian Duolingo (complete checkpoint 5) (70%) – no, haven't been doing much Duolingo
Overall, my predictions were overconfident and I underestimated the difficulty of doing various things after having a kid. Calibration was not great:
60%: 2/3 true70%: 1/6 true 80-90%: 2/2
2022 goals and predictions
Resolutions:
Author or coauthor 4 or more AI safety writeups (2 last year) (70%)Meditate on at least 230 days (196 last year) (70%)At least 450 deep work hours (306 last year) (70%)Do 3 consecutive chinups (1 last year) (60%)Avoid processed sugar at least 6 months of the year (1 month last year) (60%)
Predictions:
I will not catch covid this year (60%)I will write at least 3 blog posts (2 last year) (60%)I will read at least 5 books (4 last year) (70%)Daniel will be potty-trained by the end of the year (out of diapers when awake) (70%)
Past new year reviews: 2020-21, 2019-20, 2018-19, 2017-18, 2016-17, 2015-16, 2014-15.

2021-22 New Year review


Reflections on the first year of parenting

The first year after having a baby went by really fast – happy birthday Daniel! This post is a reflection on our experience and what we learned in the first year.

Grandparents. We were very fortunate to get a lot of help from Daniel's grandparents. My mom stayed with us when he was 1 week – 3 months old, and Janos's dad was around when he was 4-6 months old (they made it to the UK from Canada despite the pandemic). We also spent the summer in Canada with the grandparents taking care of the baby while we worked remotely.
We learned a lot about baby care from them, including nursery rhymes in our respective languages and a cool trick for dealing with the baby spitting up on himself without changing his outfit (you can put a dry cloth under the wet part of the outfit). I think our first year as parents would have been much harder without them.

Feeding. Daniel is a good eater, and in the first couple of months of life he was interested in little else (he often seemed to doze off after a feed just to cry for another feed five minutes later). I would spend whole evenings on the couch feeding him while watching through hours of David Attenborough nature series. Thankfully, around 2 months of age he became much more efficient at eating and went down to around 6 shorter feeds a day, which was much more manageable.
Around 4 months he started eating some solid food and went down to 4-5 feeds a day – this worked well for me going back to the office, since the midday feed could be replaced with solid food. He is not picky and eats almost everything, especially if he can see the grownups are eating it.
I am still breastfeeding him 2-3 times a day, which I originally wasn't expecting to do for this long. One thing I didn't realize about breastfeeding before having a kid is that it's not that easy to stop (physically or emotionally). I've been gradually decreasing the number of feeds but don't have concrete plans to stop in the next while. Both of us still seem to enjoy it, and he may be getting some covid antibodies through breast milk too.
Sleep. During the first two months, we had 2-3 long feeds during the night, and this was pretty tiring. I found that the degree of sleep disruption mattered more for my wellbeing than the absolute amount of sleep. In particular, I felt ok if I had at least 3 hours of uninterrupted sleep, while being woken up 1-1.5 hours after falling asleep felt pretty bad. From this perspective, the number night wakings mattered a lot: one was ok, two was bad but tolerable, three was terrible. I did not succeed at taking naps during the day and sometimes had insomnia at night, which was more frustrating than being woken up by the baby. My mom is a morning person and was fine with doing the morning shift starting around 5am – this allowed me to have a luxurious 4 hour block of uninterrupted sleep.
For the first 6 months, Daniel had a long afternoon nap outdoors (3-4 hours), usually in the backyard or on a walk in the park. At night he slept in a sidecar cot next to our bed until he was 4 months old, at which point we moved him to a separate room and he started waking up less often (usually once per night rather than twice). At around 9 months, we remembered that sleep training is a thing, and the internet said that he didn't need to feed at night anymore, so we tried some combination of letting him cry for a few minutes before attending to him and sleeping with earplugs so we mostly didn't hear him. After a week or two he seemingly caught on to the fact that there was no more food at night and stopped waking up. These days he generally sleeps through the night until 6 or 7am, which we are very grateful for.

Parental leave. We took a month of leave together and took the rest of the leave separately. One thing that I regret is going on parental leave too early – we both went on leave a week before the due date, while Daniel was born over 2 weeks overdue. As I belatedly found out, babies are usually overdue in my family, so going on leave before the due date was a waste of time (especially for Janos). Sitting around waiting for the baby was not that much fun, since we could not make plans more than day or two ahead. I wish I had spent that time working and left more leave to spend with the actual baby later on.
Afterwards I took 3 more months and Janos took 4.5 more months of leave (we used shared parental leave for this). Both of us felt a bit isolated when on leave, though having our housemates and our respective parents around helped with this. In retrospect, it seems better to spend more of the leave together in the newborn phase.
Group house. Raising a kid in a group house has been interesting and easier than we expected. Until recently we lived downstairs and all the other housemates lived upstairs (separated by the living room) so the others didn't hear the baby at night. Our housemates generally enjoy interacting with him and watching him learn new skills. He is a lot more social than most pandemic babies (as attested by various caregivers), which is probably a combination of being around a lot more people and natural temperament.
One upside of the group house is the flexibility of available space. There was an available spare room because a housemate was away for a while, and we used it to host our parents. Someone recently moved out from the small room upstairs, so we got that room for Daniel and moved into the room next to it.
One challenge with parenting in a group house is a higher total amount of variance in our lives, since the baby and our housemates are both sources of variance (e.g. people moving in and out, more stuff in shared spaces, etc), but so far this has not been too much. Overall, living in a group house still works well and we plan to continue.
Potty training. We've been following some combination of elimination communication and family tradition to get him used to using the potty from an early age, starting from about a month old. We got a "top hat" potty, meant for young babies who can't sit up yet, that goes between your knees with the baby sitting in your lap and leaning back on you. We've been pretty lazy about this and only put him on the potty 2-3 times a day.
His level of potty use was pretty nonlinear. He used it for #2 at the same time almost every day for a couple of months, then this pattern got thrown off when he started solids, but he still used it occasionally. He lost interest in the potty for a couple of months in summer, probably related to outgrowing the top hat potty but not yet being comfortable sitting on a regular one. He gradually got used to the regular potty, especially after we started giving him some treats when sitting on it (fruit puffs / melts / etc), which both served as a reward and helped him relax if he was restless. These days he uses the potty most of the time, but doesn't yet give a clear signal when he wants to go – hopefully this will happen in not too long.
Mental stuff. I found it mentally taxing to take care of a newborn with the background stress of the second wave of the pandemic. There seemed to be a direct relationship between the amount and quality of sleep and mental resilience. I often found that I was being harder on myself, as if I had some finite amount of patience and compassion and I was spending it all on Daniel.
One unexpected aspect of becoming a parent was that in addition to worrying about his safety during the day, I had nightmares during the night, usually involving him falling and me trying to catch him. The worst part was that these dreams woke me up, usually about half an hour after falling asleep. I often woke up standing, looking for the baby in random places like the closet and feeling pretty disoriented. This often woke Janos up as well, because I would cry out "where is he" or something along these lines. On two occasions I hit my leg on the bed frame while getting up in my sleep and got bruises on my leg. I don't seem to be getting out of bed in my sleep anymore (the weighted blanket probably helps), but I still have these dreams at least a few times a month. I hope this will eventually go away.
Physical recovery. Recovering from having a baby took longer than I expected, though I suspect my expectations were a bit unrealistic. I used to fantasize about doing chinups while wearing the baby (silly I know), but by the time I managed to do a chinup again (a few weeks ago), the baby weighed 11 kg, so that ship has sailed. The ab muscles below my ribs were pretty reluctant to get back to work after everything I did to them. I was excited to manage a one-minute plank on my elbows 5 months postpartum (after a lot of iterations of postnatal pilates videos on Youtube).
Overall, the first year has been quite a ride! Thanks Daniel for being a chill and awesome baby, we are lucky to have you :). Looking forward to many more exciting years together.


2020-21 New Year review

This is an annual post reviewing the last year and making resolutions and predictions for next year. 2020 brought a combination of challenges from living in a pandemic and becoming a parent. Other highlights include not getting sick, getting a broader perspective on my life through decluttering, and going back to Ukraine for the first time. (This post was written in bits and pieces over the past two months.)
2020 review
Life updates:
Janos and I had a son, Daniel, on Nov 11. He arrived almost 3 weeks later than expected (apparently he was waiting to be born on my late grandfather's birthday), and has been a great source of cuddles, sound effects and fragmented sleep ever since.
1 week old
6 weeks old
Some work things also went well this year – I had a paper accepted at NeurIPS, and was promoted to senior research scientist. Also, I did not get covid, and survived half a year of working from home (much credit goes to the great company of my housemates). Overall, a lot of things to be grateful for.

AI safety research:
Wrote a paper on avoiding side effects by considering future tasks, providing some theoretical grounding for the side effects problem, which was accepted to NeurIPS 2020. Contributed theoretical results for a project on the tampering problem and coauthored two papers: Avoiding Tampering Incentives in Deep RL via Decoupled Approval and REALab: An Embedded Perspective on Tampering. Wrote a blog post on the specification gaming problem for the DeepMind blog: Specification gaming: the flip side of AI ingenuity.
Effectiveness:
Working from home was definitely a productivity hit. I was mostly focused on urgent tasks, such as conference submissions and reviewing, and didn't get much research done. I did 366 hours of "deep work" (1.95 hours per work day) this year, compared to 551 hours (2.4 hours per work day) in 2019. This includes theory work, reading papers, writing papers and code, but not editing text or debugging. I got back into using work cycles, which added some helpful structure in the home environment.
I was very grateful to be living in a group house during the pandemic. While it was a bit tricky to have 5 people sharing the space when working from home, it was awesome to have an in-person community and not feel completely isolated from the world. It was also much easier to do nonzero exercise when I had someone to do it with, e.g. running in the park together.
Spending a lot of time at home inspired me to do a lot of decluttering. In particular, I went through all my old notes, got rid of most of them, and gathered the ones that still seem interesting and relevant (notes from rationality workshops, Hamming worksheets, reflections and so on). I put these into a binder for easy reading, and found it useful for getting a big picture sense of how my attitudes and problems have evolved over time. This has been particularly helpful during the pandemic, when my life has often felt small and repetitive.
I got a UK driving license for automatic cars, which took a surprising amount of practice given that I already had a US license. There was a lot to get used to with the left side of the road, the narrowness of the streets and frequent maneuvering – I spent a number of lessons just on getting the positioning right. I did the theory test in March and planned to take the road test in early summer, but then the driving schools closed for lockdown, and I ended up starting the lessons in July. I took the road test in September and didn't pass because of "undue hesitation" at a busy roundabout, so I had to repeat the test in October, two weeks before Daniel was due (thankfully, I passed this time and could forget about driving for a while).
Health:
Physical health has been pretty good this year. Last year I had 7 colds, while this year I was not sick at all – probably due to social distancing and taking zinc regularly. Thankfully, recovery after the birth was relatively quick, feeling mostly normal in around 2 weeks, though it will take some time to get my core muscles back online. I've been getting some back pain from lifting Daniel (now at 6kg thanks to his voracious appetite), which makes it all the more important to rebuild core strength.
The second half of the year came with pretty bad sleep – a lot of insomnia in the last trimester where I woke up at 3-4am for no discernible reason and couldn't fall asleep again, followed by fragmented sleep after Daniel was born. Living and working on 4-5 hours of sleep before I went on parental leave was surprisingly ok, probably because I was waking up on my own rather than being woken up in the middle of a sleep cycle. On the other hand, being woken up by a hungry baby definitely feels more meaningful than waking up at 3am for no reason and not being able to go back to sleep.
Rate of insomnia by month
Average hours awake at night by month
Average hours of sleep by month
This year has been pretty hard on my mental health due to a number of ways that the pandemic interacted with having a kid, and various problems that I considered solved have made a comeback lately. I spent most of the year at home without many forms of self-care, such as my usual exercise, sufficient sleep, or nice things like going to the sauna. While the birth went well, there were a lot more stressful interactions with the healthcare system than I had hoped. After that, there has been a combination of sleep deprivation, limited daylight, a mostly empty house after some housemates moved to the countryside, difficulty with meeting friends outside because of cold weather, making increasingly modest plans only to have them shot down by the ever-changing lockdown rules, and the exciting new covid strain we have in London that calls for high levels of caution and isolation. Thankfully, my mom was able to come stay with us for a few months to help with the baby, make food and keep us company.
Travel:
In January I visited Ukraine for the first time since I emigrated 17 years ago. I saw my dad and aunt, as well as my niece and her kids who live in a remote part of Canada but happened to be in Ukraine for the winter. It was an interesting experience to navigate around Kyiv – I no longer had a map of the city in my head, so I recognized some familiar places but could not recall where they are relative to each other, so this felt like visiting a new city with a lot of deja vu. I was pleasantly surprised by the large number of Georgian restaurants in Kyiv, which we made sure to frequent and were not disappointed.

In March we did a week-long meditation retreat at MAPLE in the US. We hesitated whether to go ahead with this plan given that flights might get canceled, but ultimately decided to go. The retreat was in a remote location in Vermont that seemed pretty safe from a covid perspective. I was advised to follow an equanimity practice that worked pretty well (focusing on acceptance rather than observation of things that come into my awareness).
Janos in his element
We had a peaceful week meditating among the snows, which unfortunately became less peaceful at the end when some people started coughing, so we spent the last couple of days meditating in masks and gloves, and left as soon as the retreat ended. Our flight back did get canceled, but we were rebooked on another one for free. Upon returning home, we self-isolated from our housemates and acquired covid tests, which were thankfully negative. We later learned that several people at the retreat tested positive for covid, so this was a close call.
In August we went camping in North Wales at a friend's cottage (after the first lockdown was lifted). The cottage itself was abandoned (and a bit spooky), but we could stay on the adjacent land and thus avoid crowds at the newly opened campsites. We enjoyed a lot of swimming in cold waterfalls and a much warmer Atlantic ocean. We also hiked up a nearby mountain Arenig Fawr (Daniel and I were taking it slow).
View from the summit of Arenig Fawr
In September we had a vacation in Madeira, where everyone was tested for covid on arrival at the airport, and there was no community transmission at the time. Madeira is a volcanic island that is basically one big mountain, and we had an interesting time driving around it (on the way to a hike, our car refused to go up a very steep road and we took a taxi the rest of the way). The terrain was a great combination of ocean and mountains.

We enjoyed large quantities of Portuguese food (as we soon learned, it did not come in small quantities). Our special favorites were local rock mussels called limpets – we ate enough of them to have a shell stacking competition.
The winner built a tower of 22, which collapsed before we could take a photo, so here is a tower of 13.
In December we did two short hikes near London, and verified that the basic algorithm of putting Daniel in a car seat and then in a carrier for the hike seems to work pretty well (he mostly sleeps through all this). I'm glad to be able to visit some nature during these strange times, it makes the world feel just a bit less small.
Iron age hill fort in Epping Forest
2020 prediction outcomes
Unsurprisingly, some predictions for the past year were messed up by the pandemic.
Resolutions:
Author or coauthor three or more academic papers (3 last year) (70%) – yes (3 papers)At most 12 non-research work commitments, such as speaking and organizing (10 last year) (80%) – yes (5 commitments). Easy, since a lot of events got canceled.Meditate on at least 270 days (290 last year) (80%) – no (244 days). The past month I only managed to meditate on 9 days, and this has not been good for me, so I need to do better next year. Read at least 7 books (5 last year) (70%) – yes (9 books). Human Compatible, The Precipice, Secret of our Success, Ender's Game, In the Realm of Hungry Ghosts, Watching the English, Positive Birth Book, The Gardener and the Carpenter, Raising a Secure Child.At least 700 deep work hours (551 last year) (70%) – no (366 hours). I found it much harder to do deep work at home, and was on parental leave for the last 2.5 months of the year.
Predictions:
I will write at least 5 blog posts (60%) – no (3 posts)Eating window at most 11 hours on at least 240 days (228 last year) (70%) – no (131 days), since I stopped doing intermittent fasting this yearI will visit at least 4 new cities with population over 100,000 (11 last year) (70%) – no (2 cities, Birmingham and Funchal). Much less travel than normal this year. At most 1 housemate turnover at Deep End (70%) – no (2 housemates). One housemate moved to live with parents in the countryside who would have likely stayed in London under normal circumstances. I finish a language in Duolingo (60%) – no, though made some progress on Mandarin (completed checkpoint 2)
2021 resolutions and predictions
Resolutions:
Avoid catching covid (90%)Author or coauthor three or more academic papers (3 last year) (70%)At most 7 non-research work commitments (5 last year) (80%)Meditate on at least 230 days (244 last year) (70%)At least 450 deep work hours (366 last year) (70%)Do 4 consecutive chinups (70%)
Predictions:
I will write at least 3 blog posts (3 last year) (60%)Janos and I will get vaccinated for covid by the end of June (60%)Daniel will get to meet all of his grandparents in person in 2021 (70%)I will return to avoiding processed sugar by the end of the year (60%)I will finish Hungarian Duolingo (complete checkpoint 5) (70%)
Past new year reviews: 2019-20, 2018-19, 2017-18, 2016-17, 2015-16, 2014-15.

2020-21 New Year review


Tradeoff between desirable properties for baseline choices in impact measures

(Cross-posted to the Alignment Forum. Summarized in Alignment Newsletter #108. Thanks to Carroll Wainwright, Stuart Armstrong, Rohin Shah and Alex Turner for helpful feedback on this post.)
Impact measures are auxiliary rewards for low impact on the agent's environment, used to address the problems of side effects and instrumental convergence. A key component of an impact measure is a choice of baseline state: a reference point relative to which impact is measured. Commonly used baselines are the starting state, the initial inaction baseline (the counterfactual where the agent does nothing since the start of the episode) and the stepwise inaction baseline (the counterfactual where the agent does nothing instead of its last action). The stepwise inaction baseline is currently considered the best choice because it does not create the following bad incentives for the agent: interference with environment processes or offsetting its own actions towards the objective. This post will discuss a fundamental problem with the stepwise inaction baseline that stems from a tradeoff between different desirable properties for baseline choices, and some possible alternatives for resolving this tradeoff.

One clearly desirable property for a baseline choice is to effectively penalize high-impact effects, including delayed effects. It is well-known that the simplest form of the stepwise inaction baseline does not effectively capture delayed effects. For example, if the agent drops a vase from a high-rise building, then by the time the vase reaches the ground and breaks, the broken vase will be the default outcome. Thus, in order to penalize delayed effects, the stepwise inaction baseline is usually used in conjunction with inaction rollouts, which predict future outcomes of the inaction policy. Inaction rollouts from the current state and the stepwise baseline state are compared to identify delayed effects of the agent's actions. In the above example, the current state contains a vase in the air, so in the inaction rollout from the current state the vase will eventually reach the ground and break, while in the inaction rollout from the stepwise baseline state the vase remains intact.

While inaction rollouts are useful for penalizing delayed effects, they do not address all types of delayed effects. In particular, if the task requires setting up a delayed effect, an agent with the stepwise inaction baseline will have no incentive to undo the delayed effect. Here are some toy examples that illustrate this problem.
Door example. Suppose the agent's task is to go to the store, which requires opening the door in order to leave the house. Once the door has been opened, the effects of opening the door are part of the stepwise inaction baseline, so the agent has no incentive to close the door as it leaves.
Red light example.  Suppose the agent's task is to drive from point A to point B along a straight road, with a reward for reaching point B. To move towards point B, the agent needs to accelerate. Once the agent has accelerated, it travels at a constant speed by default, so the noop action will move the agent along the road towards point B. Along the road (s1), there is a red light and a pedestrian crossing the road. The noop action in s1 crosses the red light and hits the pedestrian (s2). To avoid this, the agent needs to deviate from the inaction policy by stopping (s4) and then accelerating (s5).

The stepwise inaction baseline will incentivize the agent to run the red light and go to s3. The inaction rollout at s0 penalizes the agent for the predicted delayed effect of running over the pedestrian when it takes the accelerating action to go to s1. The agent receives this penalty whether or not it actually ends up running the red light or not. Once the agent has reached s1, running the red light becomes the default outcome, so the agent is not penalized for doing so (and would likely be penalized for stopping). Thus, the stepwise inaction baseline gives no incentive to avoid running the red light, while the initial inaction baseline compares to s0 and thus incentivizes the agent to stop at the red light.
This problem with the stepwise baseline arises from a tradeoff between penalizing delayed effects and avoiding offsetting incentives. The stepwise structure that makes it effective at avoiding offsetting makes it less effective at penalizing delayed effects. While delayed effects are undesirable, undoing the agent's actions is not necessarily bad. In the red light example, the action of stopping at the red light is offsetting the accelerating action. Thus, offsetting can be necessary for avoiding delayed effects while completing the task.
Whether offsetting an effect is desirable depends on whether this effect is part of the task objective. In the door-opening example, the action of opening the door is instrumental for going to the store, and many of its effects (e.g. strangers entering the house through the open door) are not part of the objective, so it is desirable for the agent to undo this action. In the vase environment shown below, the task objective is to prevent the vase from falling off the end of the belt and breaking, and the agent is rewarded for taking the vase off the belt. The effects of taking the vase off the belt are part of the objective, so it is undesirable for the agent to undo this action.
Source: Designing agent incentives to avoid side effects
The difficulty of identifying these "task effects" that are part of the objective creates a tradeoff between penalizing delayed effects and avoiding undesirable offsetting. This tradeoff can be avoided by the starting state baseline, which however produces interference incentives. The stepwise inaction baseline cannot resolve the tradeoff, since it avoids all types of offsetting, including desirable offsetting.
The initial inaction baseline can resolve this tradeoff by allowing offsetting and relying on the task reward to capture task effects and penalize the agent for offsetting them. While we cannot expect the task reward to capture what the agent should not do (unnecessary impact), capturing task effects falls under what the agent should do, so it seems reasonable to rely on the reward function for this. This would work similarly to the impact penalty penalizing all impact, and the task reward compensating for this in the case of impact that's needed to complete the task.
This can be achieved using a state-based reward function that assigns reward to all states where the task is completed. For example, in the vase environment, a state-based reward of 1 for states with an intact vase (or with vase off the belt) and 0 otherwise would remove the offsetting incentive.
If it is not feasible to use a reward function that penalizes offsetting task effects, the initial inaction baseline could be modified to avoid this kind of offsetting. If we assume that the task reward is sparse and doesn't include shaping terms, we can reset the initial state for the baseline whenever the agent receives a task reward (e.g. the reward for taking the vase off the belt in the vase environment). This results in a kind of hybrid between initial and stepwise inaction. To ensure that this hybrid baseline effectively penalizes delayed effects, we still need to use inaction rollouts at the reset and terminal states.
Another desirable property of the stepwise inaction baseline is the Markov property: it can be computed based on the previous state, independently of the path taken to that state. The initial inaction baseline is not Markovian, since it compares to the state in the initial rollout at the same time step, which requires knowing how many time steps have passed since the beginning of the episode. We could modify the initial inaction baseline to make it Markovian, e.g. by sampling a single baseline state from the inaction rollout from the initial state, or by only computing a single penalty at the initial state by comparing an agent policy rollout with the inaction rollout.
To summarize, we want a baseline to satisfy the following desirable properties: penalizing delayed effects, avoiding interference incentives, and the Markov property. We can consider avoiding offsetting incentives for task effects as a desirable property for the task reward, rather than the baseline. Assuming such a well-specified task reward, a Markovian version of the initial inaction baseline can satisfy all the criteria.


Possible takeaways from the coronavirus pandemic for slow AI takeoff

(Cross-posted to LessWrong. Summarized in Alignment Newsletter #104. Thanks to Janos Kramar for helpful feedback on this post.)
As the covid-19 pandemic unfolds, we can draw lessons from it for managing future global risks, such as other pandemics, climate change, and risks from advanced AI. In this post, I will focus on possible implications for AI risk. For a broader treatment of this question, I recommend FLI's covid-19 page that includes expert interviews on the implications of the pandemic for other types of risks.
A key element in AI risk scenarios is the speed of takeoff – whether advanced AI is developed gradually or suddenly. Paul Christiano's post on takeoff speeds defines slow takeoff in terms of the economic impact of AI as follows: "There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles." It argues that slow AI takeoff is more likely than fast takeoff, but is not necessarily easier to manage, since it poses different challenges, such as large-scale coordination. This post expands on this point by examining some parallels between the coronavirus pandemic and a slow takeoff scenario. The upsides of slow takeoff include the ability to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. I would argue that the covid-19 pandemic had these properties, but most of the world's institutions did not take advantage of them. This suggests that, unless our institutions improve, we should not expect the slow AI takeoff scenario to have a good default outcome.

Learning from experience. In the slow takeoff scenario, general AI is expected to appear in a world that has already experienced transformative change from less advanced AI, and institutions will have a chance to learn from problems with these AI systems. An analogy could be made with learning from dealing with less "advanced" epidemics like SARS that were not as successful as covid-19 at spreading across the world. While some useful lessons were learned, they were not successfully generalized to covid-19, which had somewhat different properties than these previous pathogens (such as asymptomatic transmission and higher virulence). Similarly, general AI may have somewhat different properties from less advanced AI that would make mitigation strategies more difficult to generalize. Warning signs. In the coronavirus pandemic response, there has been a lot of variance in how successfully governments acted on warning signs. Western countries had at least a month of warning while the epidemic was spreading in China, which they could have used to stock up on PPE and build up testing capacity, but most did not do so. Experts have warned about the likelihood of a coronavirus outbreak for many years, but this did not lead most governments to stock up on medical supplies. This was a failure to take cheap preventative measures in response to advance warnings about a widely recognized risk with tangible consequences, which is not a good sign for the case where the risk is less tangible and well-understood (such as risk from general AI). Consensus on the problem. During the covid-19 epidemic, the abundance of warning signs and past experience with previous pandemics created an opportunity for a timely consensus that there is a serious problem. However, it actually took a long time for a broad consensus to emerge – the virus was often dismissed as "overblown" and "just like the flu" as late as March 2020. A timely response to the risk required acting before there was a consensus, thus risking the appearance of overreacting to the problem. I think we can also expect this to happen with advanced AI. Similarly to the discussion of covid-19, there is an unfortunate irony where those who take a dismissive position on advanced AI risks are often seen as cautious, prudent skeptics, while those who advocate early action are portrayed as "panicking" and overreacting. The "moving goalposts" effect, where new advances in AI are dismissed as not real AI, could continue indefinitely as increasingly advanced AI systems are deployed. I would expect the "no fire alarm" hypothesis to hold in the slow takeoff scenario – there may not be a consensus on the importance of general AI until it arrives, so risks from advanced AI would continue to be seen as "overblown" until it is too late to address them.
We can hope that the transformative technological change involved in the slow takeoff scenario will also help create more competent institutions without these weaknesses. We might expect that institutions unable to adapt to the fast pace of change will be replaced by more competent ones. However, we could also see an increasingly chaotic world where institutions fail to adapt without better institutions being formed quickly enough to replace them. Success in the slow takeoff scenario depends on institutional competence and large-scale coordination. Unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff.


2019-20 New Year review

This is an annual post reviewing the last year and making resolutions and predictions for next year. This year's edition features sleep tracking, intermittent fasting, overcommitment busting, and evaluating calibration for all annual predictions since 2014.
2019 review
AI safety research:

Wrote an updated version of the relative reachability paper, including an ablation study on design choices.
Coauthored a paper on modeling AGI Safety frameworks with causal influence diagrams, accepted to IJCAI workshop.
Wrote a paper on avoiding side effects by considering future tasks, accepted to NeurIPS workshop.
Co-ran a subteam of the safety team focusing on agent incentive design (ongoing).

AI safety outreach:

Co-organized FLI's Beneficial AGI conference in Puerto Rico, a more long-term focused sequel to the original Puerto Rico conference and the Asilomar conference. This year I was the program chair for the technical safety track of the conference.
Co-organized the ICLR AI safety workshop, Safe Machine Learning: Specification, Robustness and Assurance. This was my first time running a paper reviewing process.
Gave a talk at the IJCAI AI safety workshop on specification, robustness an assurance problems.
Took part in the DeepMind podcast episode on AI safety ("I, robot").

Work effectiveness:

At the beginning of the year, I found myself overcommitted and kind of burned out. My previous efforts to reduce overcommitment had proved insufficient to not feel stressed and overwhelmed most of the time.
In February, I made a rule for myself to decline all non-research commitments that don't seem like exceptional opportunities. The form that I made last year for evaluating commitments (which I have to fill out before accepting anything) has been helpful for enforcing this rule and avoiding impulsive decisions. The number of commitments went down from 24 in 2018 to 10 in 2019. This has been working well in terms of having more time for research and feeling better about life.
Organizing a conference and a workshop back to back was a bit much, and I feel done with organizing large events for a while.

Stopped using work cycles and pomodoros since I've been finding the structure a bit too restrictive. Might resume at some point.
Hours of "deep work" per month, as defined in the Deep Work book. This includes things like thinking about research problems, coding, reading and writing papers. It does not include email, organizing, most meetings, coding logistics (e.g. setup or running experiments), etc.

For comparison, deep work hours from 2018. My definition of deep work has somewhat broadened over time, but not enough to account for this difference.

Health / self-care:

I had 7 colds this past year, which is a lot more than my usual rate of 1-2 per year. The first three were in Jan-Feb, which seemed related to the overcommitment burnout. Hopefully supplementing zinc will help.
Averaged 7.2 hours of sleep per night, excluding jetlag (compared to 6.9 hours in 2018).
About a 10% rate of insomnia (excluding jetlag), similar to the end of last year.
Tried the Oura ring and the Dreem headband for measuring sleep quality. The Oura ring consistently thinks I wake up many times per night (probably because I move around a lot) and predicts less than half an hour each of deep and REM sleep. The Dreem, which actually measures EEG signals, estimates that I get an average of 1.3 hours of deep sleep and 1.8 hours of REM sleep per night, which is more than I expected.
Started a relaxed form of intermittent fasting in March (aiming for a 10 hour eating window), mostly for longevity and to improve my circadian rhythm. My average eating window length over the year was 10.5 hours, so I wasn't very strict about it (mostly just avoiding snacks after dinner). One surprising thing I learned was that I can fall asleep just fine while hungry, and am often less hungry when I wake up. My average hours of sleep went up from 6.96 in the 6 months before starting intermittent fasting to 7.32 in the 6 months after. I went to sleep 44 minutes earlier and waking up 20 minutes earlier on average, though the variance of my bedtime actually went up a bit. Overall it seems plausibly useful easy enough to continue next year.

Fun stuff:

Did a Caucasus hiking trek in Georgia with family, and consumed a lot of wild berries and hazelnuts along the way.

Did a road trip in southern Iceland (also with family), saw a ridiculous number of pretty waterfalls, and was in the same room with (artificial) lava.

Took an advanced class in aerial silks for the first time (I felt a bit underqualified, but learned a lot of fun moves).
Ran a half-marathon along the coast in Devon on hilly terrain in 3 hours and 23 minutes.
Made some progress on handstands in yoga class (can hold it away from the wall for a few seconds)
Did two circling retreats (relational meditation)
Read books: The Divide, 21 Lessons for the 21st Century, The Circadian Code, So Good They Can't Ignore You, Ending Aging (skimmed).
Got into Duolingo (brushed up my Spanish and learned a bit of Mandarin). Currently in a quasi-competition with Janos for studying each other's languages.

2019 prediction outcomes
Resolutions:

Author or coauthor two or more academic papers (50%) – yes (3 papers)
Accept at most 17 non-research commitments (24 last year) (60%) – yes (10 commitments)
Meditate on at least 250 days (60%) – yes (290 days)

Predictions:

Relative reachability paper accepted at a major conference, not counting workshops (60%) – no
Continue avoiding processed sugar for the next year (85%) – no (still have the intention and mostly follow it, but less strictly / consistently)
1-2 housemate turnover at Deep End (2 last year) (80%) – yes (1 housemate moved in)
At least 5 rationality sessions will be hosted at Deep End (80%) – no

Calibration over all annual reviews:

50-70% well-calibrated, 80-90% overconfident (66 predictions total)
Calibration is generally better in 2017-19 (23 predictions) than in 2014-16 (43 predictions). There were only 3 70% predictions in 2017-19, so the 100% accuracy is noisy.
Unsurprisingly, resolutions are more often correct than other predictions (72% vs 56% correct)

2020 resolutions and predictions
Resolutions

Author or coauthor three or more academic papers (3 last year) (70%)
At most 12 non-research commitments (10 last year) (80%)
Meditate on at least 270 days (290 last year) (80%)
Read at least 7 books (5 last year) (70%)
At least 700 deep work hours (551 last year) (70%)

Predictions

I will write at least 5 blog posts (60%)
Eating window at most 11 hours on at least 240 days (228 last year) (70%)
I will visit at least 4 new cities with population over 100,000 (11 last year) (70%)
At most 1 housemate turnover at Deep End (70%)
I finish a language in Duolingo (60%)

Past new year reviews: 2018-19, 2017-18, 2016-17, 2015-16, 2014-15.

2019-20 New Year review


Retrospective on the specification gaming examples list

My post about the specification gaming list was recently nominated for the LessWrong 2018 Review (sort of like a test of time award), which prompted me to write a retrospective (cross-posted here).
I've been pleasantly surprised by how much this resource has caught on in terms of people using it and referring to it (definitely more than I expected when I made it). There were 30 examples on the list when was posted in April 2018, and 20 new examples have been contributed through the form since then.  I think the list has several properties that contributed to wide adoption: it's fun, standardized, up-to-date, comprehensive, and collaborative.
Some of the appeal is that it's fun to read about AI cheating at tasks in unexpected ways (I've seen a lot of people post on Twitter about their favorite examples from the list). The standardized spreadsheet format seems easier to refer to as well. I think the crowdsourcing aspect is also helpful – this helps keep it current and comprehensive, and people can feel some ownership of the list since can personally contribute to it. My overall takeaway from this is that safety outreach tools are more likely to be impactful if they are fun and easy for people to engage with.
This list had a surprising amount of impact relative to how little work it took me to put it together and maintain it. The hard work of finding and summarizing the examples was done by the people putting together the lists that the master list draws on (Gwern, Lehman, Olsson, Irpan, and others), as well as the people who submit examples through the form. What I do is put them together in a common format and clarify and/or shorten some of the summaries. I also curate the examples to determine whether they fit the definition of specification gaming (as opposed to simply a surprising behavior or solution). Overall, I've probably spent around 10 hours so far on creating and maintaining the list, which is not very much. This makes me wonder if there is other low hanging fruit in the safety resources space that we haven't picked yet.
I have been using it both as an outreach and research tool. On the outreach side, the resource has been helpful for making the argument that safety problems are hard and need general solutions, by making it salient just in how many ways things could go wrong. When presented with an individual example of specification gaming, people often have a default reaction of "well, you can just close the loophole like this". It's easier to see that this approach does not scale when presented with 50 examples of gaming behaviors. Any given loophole can seem obvious in hindsight, but 50 loopholes are much less so. I've found this useful for communicating a sense of the difficulty and importance of Goodhart's Law.
On the research side, the examples have been helpful for trying to clarify the distinction between reward gaming and tampering problems. Reward gaming happens when the reward function is designed incorrectly (so the agent is gaming the design specification), while reward tampering happens when the reward function is implemented incorrectly or embedded in the environment (and so can be thought of as gaming the implementation specification). The boat race example is reward gaming, since the score function was defined incorrectly, while the Qbert agent finding a bug that makes the platforms blink and gives the agent millions of points is reward tampering. We don't currently have any real examples of the agent gaining control of the reward channel (probably because the action spaces of present-day agents are too limited), which seems qualitatively different from the numerous examples of agents exploiting implementation bugs.
I'm curious what people find the list useful for – as a safety outreach tool, a research tool or intuition pump, or something else? I'd also be interested in suggestions for improving the list (formatting, categorizing, etc). Thanks everyone who has contributed to the resource so far!


Classifying specification problems as variants of Goodhart's Law

(Coauthored with Ramana Kumar and cross-posted from the Alignment Forum. Summarized in Alignment Newsletter #76.)
There are a few different classifications of safety problems, including the Specification, Robustness and Assurance (SRA) taxonomy and the Goodhart's Law taxonomy. In SRA, the specification category is about defining the purpose of the system, i.e. specifying its incentives.  Since incentive problems can be seen as manifestations of Goodhart's Law, we explore how the specification category of the SRA taxonomy maps to the Goodhart taxonomy. The mapping is an attempt to integrate different breakdowns of the safety problem space into a coherent whole. We hope that a consistent classification of current safety problems will help develop solutions that are effective for entire classes of problems, including future problems that have not yet been identified.
The SRA taxonomy defines three different types of specifications of the agent's objective: ideal (a perfect description of the wishes of the human designer), design (the stated objective of the agent) and revealed (the objective recovered from the agent's behavior). It then divides specification problems into design problems (e.g. side effects) that correspond to a difference between the ideal and design specifications, and emergent problems (e.g. tampering) that correspond to a difference between the design and revealed specifications.
In the Goodhart taxonomy, there is a variable U* representing the true objective, and a variable U representing the proxy for the objective (e.g. a reward function). The taxonomy identifies four types of Goodhart effects: regressional (maximizing U also selects for the difference between U and U*), extremal (maximizing U takes the agent outside the region where U and U* are correlated), causal (the agent intervenes to maximize U in a way that does not affect U*), and adversarial (the agent has a different goal W and exploits the proxy U to maximize W).
We think there is a correspondence between these taxonomies: design problems are regressional and extremal Goodhart effects, while emergent problems are causal Goodhart effects. The rest of this post will explain and refine this correspondence.

The SRA taxonomy needs to be refined in order to capture the distinction between regressional and extremal Goodhart effects, and to pinpoint the source of causal Goodhart effects. To this end, we add a model specification as an intermediate point between the ideal and design specifications, and an implementation specification between the design and revealed specifications.
The model specification is the best proxy within a chosen formalism (e.g. model class or specification language), i.e. the proxy that most closely approximates the ideal specification. In a reinforcement learning setting, the model specification is the reward function (defined in the given MDP/R over the given state space) that best captures the human designer's preferences.

The ideal-model gap corresponds to the model design problem (regressional Goodhart): choosing a model that is tractable but also expressive enough to approximate the ideal specification well.
The model-design gap corresponds to proxy design problems (extremal Goodhart), such as specification gaming and side effects.

While the design specification is a high-level description of what should be executed by the system, the implementation specification is a specification that can be executed, which includes agent and environment code (e.g. an executable Linux binary). (We note that it is also possible to define other specification levels at intermediate levels of abstraction between design and implementation, e.g. using pseudocode rather than executable code.)

The design-implementation gap corresponds to tampering problems (causal Goodhart), since they exploit implementation flaws (such as bugs that allow the agent to overwrite the reward). (Note that tampering problems are referred to as wireheading and delusions in the SRA.)
The implementation-revealed gap corresponds to robustness problems in the SRA (e.g. unsafe exploration).

In the model design problem, U is the best approximation of U* within the given model.  As long as the global maximum M for U is not exactly the same as the global maximum M* for U*, the agent will not find M*. This corresponds to regressional Goodhart: selecting for U will also select for the difference between U and U*, so the optimization process will overfit to U at the expense of U*.

In proxy design problems, U and U* are correlated under normal circumstances, but the correlation breaks in situations when U is maximized, which is an extremal Goodhart effect. The proxy U is often designed to approximate U* by having a maximum at a global maximum M* of U*. Different ways that this approximation fails produce different problems.

In specification gaming problems, M* turns out to be a local (rather than global) maximum for U, e.g. if M* is the strategy of following the racetrack in the boat race game. The agent finds the global maximum M for U, e.g. the strategy of going in circles and repeatedly hitting the same reward blocks. This is an extrapolation of the reward function outside the training domain that it was designed for, so the correlation with the true objective no longer holds. This is an extremal Goodhart effect due to regime change.

In side effect problems, M* is a global maximum for U, but U incorrectly approximates U* by being flat in certain dimensions (corresponding to indifference to certain variables, e.g. whether a vase is broken). Then the set of global maxima for U is much larger than the set of global maxima for U*, and most points in that set are not global maxima for U*. Maximizing U can take the agent into a region where U doesn't match U*, and the agent finds a point M that is also a global maximum for U, but not a global maximum for U*. This is an extremal Goodhart effect due to model insufficiency.

Current solutions to proxy design problems involve taking the proxy less literally: by injecting uncertainty (e.g. quantilization), avoiding extrapolation (e.g. inverse reward design), or adding a term for omitted preferences (e.g. impact measures).
In tampering problems, we have a causal link U* -> U. Tampering occurs when the agent intervenes on some variable W that has a causal effect on U that does not involve U*, which is a causal Goodhart effect. W could be the reward function parameters, the human feedback data (in reward learning), the observation function parameters (in a POMDP), or the status of the shutdown button. The overall structure is  U* -> U <- W.
For example, in the Rocks & Diamonds environment, U* is the number of diamonds delivered by the agent to the goal area. Intervening on the reward function to make it reward rocks increases the reward U without increasing U* (the number of diamonds delivered).
Current solutions to tampering problems involve modifying the causal graph to remove the tampering incentives, e.g. by using approval-direction or introducing counterfactual variables.
[Updated] We think that mesa-optimization belongs in the implementation-revealed gap, rather than in the design-implementation gap, since it can happen during the learning process even if the implementation specification matches the ideal specification, and can be seen as a robustness problem. When we consider this problem one level down, as a specification problem for the mesa-optimizer from the main agent's perspective, it can take the form of any of the four Goodhart effects. The four types of alignment problems in the mesa-optimization paper can be mapped to the four types of Goodhart's Law as follows: approximate alignment is regressional, side effect alignment is extremal, instrumental alignment is causal, and deceptive alignment is adversarial.
This correspondence is consistent with the connection between the Goodhart taxonomy and the selection vs control distinction, where regressional and extremal Goodhart are more relevant for selection, while causal Goodhart is more relevant for control. The design specification is generated by a selection process, while the revealed specification is generated by a control process. Thus, design problems represent difficulties with selection, while emergent problems represent difficulties with control.
[Updated] Putting it all together:

In terms of the limitations of this mapping, we are not sure about model specification being the dividing line between regressional and extremal Goodhart. For example, a poor choice of model specification could deviate from the ideal specification in systematic ways that result in extremal Goodhart effects. It is also unclear how adversarial Goodhart fits into this mapping. Since an adversary can exploit any differences between U* and U (taking advantage of the other three types of Goodhart effects) it seems that adversarial Goodhart effects can happen anywhere in the ideal-implementation gap.
We hope that you find the mapping useful for your thinking about the safety problem space, and welcome your feedback and comments. We are particularly interested if you think some of the correspondences in this post are wrong.
(Thanks to Jan Leike and Tom Everitt for their helpful feedback on this post.)

Classifying specification problems as variants of Goodhart’s Law


ICLR Safe ML workshop report

This year the ICLR conference hosted topic-based workshops for the first time (as opposed to a single track for workshop papers), and I co-organized the Safe ML workshop. One of the main goals was to bring together near and long term safety research communities.

The workshop was structured according to a taxonomy that incorporates both near and long term safety research into three areas – specification, robustness and assurance.

Specification: define the purpose of the system
Robustness: design system to withstand perturbations
Assurance: monitor and control system activity

Reward hacking
Side effects
Preference learning
Fairness

Adaptation
Verification
Worst-case robustness
Safe exploration

Interpretability
Monitoring
Privacy
Interruptibility

We had an invited talk and a contributed talk in each of the three areas.

Talks
In the specification area, Dylan Hadfield-Menell spoke about formalizing the value alignment problem in the Inverse RL framework.

David Krueger presented a paper on hidden incentives for the agent to shift its task distribution in the meta-learning setting.

In the robustness area, Ian Goodfellow argued for dynamic defenses against adversarial examples and encouraged the research community to consider threat models beyond small perturbations within a norm ball of the original data point.

Avraham Ruderman presented a paper on worst-case analysis for discovering surprising behaviors (e.g. failing to find the goal in simple mazes).

In the assurance area, Cynthia Rudin argued that interpretability doesn't have to trade off with accuracy (especially in applications), and that it is helpful for solving research problems in all areas of safety.

Beomsu Kim presented a paper explaining why adversarial training improves the interpretability of gradients for deep neural networks.

Panels
The workshop panels discussed possible overlaps between different research areas in safety and research priorities going forward.
In terms of overlaps, the main takeaway was that advancing interpretability is useful for all safety problems. Also, adversarial robustness can contribute to value alignment – e.g. reward gaming behaviors can be viewed as a system finding adversarial examples for its reward function. However, there was a cautionary point that while near- and long-term problems are often similar, solutions might not transfer well between these areas (e.g. some solutions to near-term problems might not be sufficiently general to help with value alignment).

The research priorities panel recommended more work on adversarial examples with realistic threat models (as mentioned above), complex environments for testing value alignment (e.g. creating new structures in Minecraft without touching existing ones), fairness formalizations with more input from social scientists, and improving cybersecurity.
Papers
Out of the 35 accepted papers, 5 were on long-term safety / value alignment, and the rest were on near-term safety. Half of the near-term paper submissions were on adversarial examples, so the resulting pool of accepted papers was skewed as well: 14 on adversarial examples, 5 on interpretability, 3 on safe RL, 3 on other robustness, 2 on fairness, 2 on verification, and 1 on privacy. Here is a summary of the value alignment papers:
Misleading meta-objectives and hidden incentives for distributional shift by Krueger et al shows that RL agents in a meta-learning context have an incentive to shift their task distribution instead of solving the intended task. For example, a household robot whose task is to predict whether its owner will want coffee could wake up its owner early in the morning to make this prediction task easier. This is called a 'self-induced distributional shift' (SIDS), and the incentive to do so is a 'hidden incentive for distributional shift' (HIDS). The paper demonstrates this behavior experimentally and shows how to avoid it.

How useful is quantilization for mitigating specification-gaming? by Ryan Carey introduces variants of several classic environments (Mountain Car, Hopper and Video Pinball) where the observed reward differs from the true reward, creating an opportunity for the agent to game the specification of the observed reward. The paper shows that a quantilizing agent avoids specification gaming and performs better in terms of true reward than both imitation learning and a regular RL agent on all the environments.

Delegative Reinforcement Learning: learning to avoid traps with a little help by Vanessa Kosoy introduces an RL algorithm that avoids traps in the environment (states where regret is linear) by delegating some actions to an external advisor, and achieves sublinear regret in a continual learning setting. (Summarized in Alignment Newsletter #57)

Generalizing from a few environments in safety-critical reinforcement learning by Kenton et al investigates how well RL agents avoid catastrophes in new gridworld environments depending on the number of training environments. They find that both model ensembling and learning a catastrophe classifier (used to block actions) are helpful for avoiding catastrophes, with different safety-performance tradeoffs on new environments.

Regulatory markets for AI safety by Clark and Hadfield proposes a new model for regulating AI development where regulation targets are required to choose regulatory services from a private market that is overseen by the government. This allows regulation to efficiently operate on a global scale and keep up with the pace of technological development and better ensure safe deployment of AI systems. (Summarized in Alignment Newsletter #55)

The workshop got a pretty good turnout (around 100 people). Thanks everyone for participating, and thanks to our reviewers, sponsors, and my fellow organizers for making it happen!

(Cross-posted to the FLI blog. Thanks to Janos Kramar for his feedback on this post.)

ICLR Safe ML workshop report


2018-19 New Year review

2018 progress
Research / AI safety:

Wrote a paper on measuring side effects using relative reachability in May, and presented the results at the ICML GoalsRL workshop and the AI safety summer school. Since then, some new approaches have come out using my method as a baseline :).
Made a list of 30 specification gaming examples in AI (assembled from several existing lists). Since the list was posted in April, 16 new examples have been contributed through the form (thanks everyone!). The list received some attention on Twitter, and I was interviewed about it by Wired and the Times.
Was in the top 30% of NeurIPS reviewers.
Gave talks at the Oxford AI Society, EA Global London, etc.
Got involved in organizing the upcoming ICLR AI safety workshop, Safe Machine Learning: Specification, Robustness and Assurance.

Rationality / effectiveness:

Attended the CFAR mentoring workshop in Prague, and started running rationality training sessions with Janos at our group house.
Started using work cycles – focused work blocks (e.g. pomodoros) with built-in reflection prompts. I think this has increased my productivity and focus to some degree. The prompt "how will I get started?" has been surprisingly helpful given its simplicity.
Stopped eating processed sugar for health reasons at the end of 2017 and have been avoiding it ever since.

This has been surprisingly easy, especially compared to my earlier attempts to eat less sugar. I think there are two factors behind this: avoiding sugar made everything taste sweeter (so many things that used to taste good now seem inedibly sweet), and the mindset shift from "this is a luxury that I shouldn't indulge in" to "this is not food".
Unfortunately, I can't make any conclusions about the effects on my mood variables because of some issues with my data recording process :(.

Declining levels of insomnia (excluding jetlag):

22% of nights in the first half of 2017, 16% in the second half of 2017, 16% in the first half of 2018, 10% in the second half of 2018.
This is probably an effect of the sleep CBT program I did in 2017, though avoiding sugar might be a factor as well.

Made some progress on reducing non-research commitments (talks, reviewing, organizing, etc).

Set up some systems for this: a spreadsheet to keep track of requests to do things (with 0-3 ratings for workload and 0-2 ratings for regret) and a form to fill out whenever I'm thinking of accepting a commitment.
My overall acceptance rate for commitments has gone down a bit from 29% in 2017 to 24% in 2018. The average regret per commitment went down from 0.66 in 2017 to 0.53 in 2018.
However, since the number of requests has gone up, I ended up with more things to do overall: 12 commitments with a total of 23 units of workload in 2017 vs 19 commitments with a total of 33 units of workload in 2018. (1 unit of workload ~ 5 hours)

Fun stuff:

Hiked in the Alps for the first time:

Tour de Mont Blanc – a weeklong hike around Mont Blanc going through France, Italy and Switzerland. It felt funny to cross a mountain pass and end up in a different country without anyone checking my passport. There were a lot of meadows and cows.
Monte Rosa glacier hike (Gnifetti normal route). We were all connected by a rope in case someone falls into a crack in the ice. The first night (at 3500m) I could not sleep at all due to altitude and had the interesting experience of a full day hike afterwards.

Spontaneous solo trip to Amsterdam for my birthday
Helped run the Dead Hand Path camp at Burning Man and organized a series of AI safety talks
Read the Book of Why, Other Minds, The Player of Games, Life 3.0, The Elephant in the Brain.
Did 5 chinups in a row (only once, usually I can do 3)
Learned a headstand in yoga class
Learned some new moves in aerial silks
Our group house has been adopted by a neighbour's cat (it all started with crashing parties). After some of our housemates moved a few blocks away, the cat has been splitting her time between the two houses.

2018 prediction outcomes
Resolutions:

Write at least 2 AI blog posts that are not about conferences (1 last year) (70%) – 4 posts
Avoid processed sugar at least until end of March (90%) – yes (still going)
Do at most 4 non-research talks/panels (7 last year) (50%) – 5 talks
Meditate on at least 250 days (50%) – 283 days

Predictions:

Our AI safety team will have at least two papers accepted for publication at a major conference, not counting workshops (80%) – yes
I will write at least 6 blog posts (60%) – wrote 5 posts
I will go to at least 100 exercise classes (80 last year) (60%) – 123 classes
1-2 housemate turnover at the Deep End (3 last year) (70%) – 2 housemates
I will visit at least 3 new cities with population over 100,000 (4 last year) (50%) – Amsterdam, Geneva, Stockholm, Prague
I will go on at least 2 hikes (4 last year) (90%) – 3 major hikes (Vancouver Island, Tour de Mont Blanc, Monte Rosa)

Calibration:

50-60%: 3 correct, 2 wrong
70-90%: 5 correct
High-confidence predictions are underconfident, low-confidence predictions are well-calibrated.

2019 goals and predictions
Resolutions:

Author or coauthor two or more academic papers (50%)
Accept at most 17 non-research commitments (24 last year) (60%)
Meditate on at least 250 days (60%)

Predictions:

Relative reachability paper accepted at a major conference, not counting workshops (60%)
Continue avoiding processed sugar for the next year (85%)
1-2 housemate turnover at Deep End (2 last year) (80%)
At least 5 rationality sessions will be hosted at Deep End (80%)

Past new year reviews: 2017-18, 2016-17, 2015-16, 2014-15.

2018-19 New Year review