OpenAI unconference on machine learning

Last weekend, I attended OpenAI’s self-organizing conference on machine learning (SOCML 2016), meta-organized by Ian Goodfellow (thanks Ian!). It was held at OpenAI’s new office, with several floors of large open spaces. The unconference format was intended to encourage people to present current ideas alongside with completed work. The schedule mostly consisted of 2-hour blocks with broad topics like “reinforcement learning” and “generative models”, guided by volunteer moderators. I especially enjoyed the sessions on neuroscience and AI and transfer learning, which had smaller and more manageable groups than the crowded popular sessions, and diligent moderators who wrote down the important points on the whiteboard. Overall, I had more interesting conversation but also more auditory overload at SOCML than at other conferences.
To my excitement, there was a block for AI safety along with the other topics. The safety session became a broad introductory Q&A, moderated by Nate Soares, Jelena Luketina and me. Some topics that came up: value alignment, interpretability, adversarial examples, weaponization of AI.
AI safety discussion group (image courtesy of Been Kim)

One value alignment question was how to incorporate a diverse set of values that represents all of humanity in the AI’s objective function. We pointed out that there are two complementary problems: 1) getting the AI’s values to be in the small part of values-space that’s human-compatible, and 2) averaging over that space in a representative way. People generally focus on the ways in which human values differ from each other, which leads them to underestimate the difficulty of the first problem and overestimate the difficulty of the second. We also agreed on the importance of allowing for moral progress by not locking in the values of AI systems.
Nate mentioned some alternatives to goal-optimizing agents – quantilizers and approval-directed agents. We also discussed the limitations of using blacklisting/whitelisting in the AI’s objective function: blacklisting is vulnerable to unforeseen shortcuts and usually doesn’t work from a security perspective, and whitelisting hampers the system’s ability to come up with creative solutions (e.g. the controversial move 37 by AlphaGo in the second game against Sedol).
Been Kim brought up the recent EU regulation on the right to explanation for algorithmic decisions. This seems easy to game due to lack of good metrics for explanations. One proposed metric was that a human would be able to predict future model outputs from the explanation. This might fail for better-than-human systems by penalizing creative solutions if applied globally, but seems promising as a local heuristic.
Ian Goodfellow mentioned the difficulties posed by adversarial examples: an imperceptible adversarial perturbation to an image can make a convolutional network misclassify it with very high confidence. There might be some kind of No Free Lunch theorem where making a system more resistant to adversarial examples would trade off with performance on non-adversarial data.
We also talked about dual-use AI technologies, e.g. advances in deep reinforcement learning for robotics that could end up being used for military purposes. It was unclear whether corporations or governments are more trustworthy with using these technologies ethically: corporations have a profit motive, while governments are more likely to weaponize the technology.

More detailed notes by Janos coming soon! For a detailed overview of technical AI safety research areas, I highly recommend reading Concrete Problems in AI Safety.
Cross-posted to the FLI blog.

OpenAI unconference on machine learning


Looking back at my grad school journey

I recently defended my PhD thesis, and a chapter of my life has now come to an end. It feels both exciting and a bit disorienting to be done with this phase of much stress and growth. My past self who started this five years ago, with a very vague idea of what she was getting into, was a rather different person from my current self.
I have developed various skills over these five years, both professionally and otherwise. I learned to read papers and explain them to others, to work on problems that take months rather than hours and be content with small bits of progress. I used to believe that I should be interested in everything, and gradually gave myself permission not to care about most topics to be able to focus on things that are actually interesting to me, developing some sense of discernment. In 2012 I was afraid to comment on the LessWrong forum because I might say something stupid and get downvoted – in 2013 I wrote my first post, and in 2014 I started this blog. I went through the Toastmasters program and learned to speak in front of groups, though I still feel nervous when speaking on technical topics, especially about my own work. I co-founded a group house and a nonprofit, both of which are still flourishing. I learned how to run events and lead organizations, starting with LessWrong meetups and the Harvard Toastmasters club, which were later displaced by running FLI.

I remember agonizing over whether I should do a PhD or not, and I wish I had instead spent more time deciding where and how to do it. I applied to a few statistics departments in the Boston area and joined the same department that Janos was in, without seriously considering computer science, even though my only research experience back in undergrad was in that field. The statistics department was full of interesting courses and brilliant people that taught me a great deal, but the cultural fit wasn’t quite right and I felt a bit out of place there. I eventually found my way to the computer science department at the end of my fourth year, but I wish I had started out there to begin with.
My research work took a rather meandering path that somehow came together in the end. My first project was part of the astrostatistics seminar, which I was not particularly motivated about, but I expected myself to be interested in everything. I never quite understood what people were talking about in the seminar or what I was supposed to be doing, and quietly dropped the project at the end of my first year when leaving for my quantitative analyst internship at D.E.Shaw. The internship was my first experience in industry, where I learned factor analysis and statistical coding in Python (the final review from my manager boiled down to “great coder, research skills need work”). In second year, my advisor offered me a project that was unfinished by his previous students, which would take a few months to polish up. The project was on a new method for classification and variable selection called SBFC. I dug up a bunch of issues with the existing model and code, from runtime performance to MCMC detailed balance, and ended up stuck on the project for 3 years. During that time, I dabbled with another project that sort of petered out, did a Google internship on modeling ad quality, and sank a ton of time into FLI. In the middle of fourth year, SBFC was still my only project, and things were not looking great for graduating.
This was when I realized that the part of statistics that was interesting to me was the overlap with computer science and AI, a.k.a. machine learning. I went to the NIPS conference for the first time, and met a lot of AI researchers – I didn’t understand a lot of their work, but I liked the way they thought. I co-organized FLI’s Puerto Rico conference and met more AI people there. I finally ventured outside the stats department and started sitting in on ML lab meetings at the CS department, which mostly consisted of research updates on variational autoencoders that went right over my head. I studied a lot to fill the gaps in my ML knowledge that were not covered by my statistics background, namely neural networks and reinforcement learning (still need to read Sutton & Barto…). To my surprise, many people at the ML lab were also transplants from other departments, officially doing PhDs in math or physics.
That summer I did my second internship at Google, on sum-product network models (SPNs) for anomaly detection in the Knowledge Graph. I wondered if it would result in a paper that could be shoehorned into my thesis, and whether I could find a common thread between SPNs, SBFC and my upcoming project at the ML lab. This unifying theme turned out to be interpretability – the main selling point of SBFC, an advantage of SPNs over other similarly expressive models, and one of my CS advisor’s interests. Working on interpretability was a way to bring more of the statistical perspective into machine learning, and seemed relevant to AI safety as well. With this newfound sense of direction, in a new environment, my fifth year had as much research output as the previous three put together, and I presented two workshop posters in 2016 – on SPNs at ICLR, and on RNN interpretability at ICML.
Volunteering for FLI during grad school started out as a kind of double life, and ended up interacting with my career in interesting ways. For a while I didn’t tell anyone in my department that I co-founded a nonprofit trying to save the world from existential risk, which was often taking up more of my time than research. However, FLI’s outreach work on AI safety was also beneficial to me – as one of the AI experts on the FLI core team, I met a lot of AI researchers who I may not have connected with otherwise. When I met the DeepMind founders at the Puerto Rico conference, I would not have predicted that I’ll be interviewing for their AI safety team a year later. The two streams of my interests, ML and AI safety, have finally crossed, and the double life is no more.
What lessons have I drawn from the grad school experience, and what advice could I give to others?

Going to conferences and socializing with other researchers was super useful and fun. I highly recommend attending NIPS and ICML even if you’re not presenting.
Academic departments vary widely in their requirements. For example, the statistics department expected PhD students to teach 10 sections (I got away with doing 5 sections and it was still a lot of work), while the CS department only expected 1-2 sections.
Internships were a great source of research experience and funding (a better use of time than teaching, in my opinion). It’s worth spending a summer interning at a good company, even if you are definitely going into academia.
Contrary to common experience, writer’s block was not an obstacle for me. My actual bottleneck was coding, debugging and running experiments, which was often tedious and took over half of my research time, so it’s well worth optimizing those aspects of the work.
The way FLI ended up contributing to my career path reminds me of a story about Steve Jobs sitting in on a calligraphy class that later turned out to be super relevant to creating snazzy fonts for Apple computers. I would recommend making time for seemingly orthogonal activities during grad school that you’re passionate about, both because they provide a stimulating break from research, and because they could become unexpectedly useful later.

Doing a PhD was pretty stressful for me, but ultimately worthwhile. A huge thank you to everyone who guided and supported me through it!

Looking back at my grad school journey


Highlights from the Deep Learning Summer School

A few weeks ago, Janos and I attended the Deep Learning Summer School at the University of Montreal. Various well-known researchers covered topics related to deep learning, from reinforcement learning to computational neuroscience (see the list of speakers with slides and videos). Here are a few ideas that I found interesting in the talks (this list is far from exhaustive):
Cross-modal learning (Antonio Torralba)
You can do transfer learning in convolutional neural nets by freezing the parameters in some layers and retraining others on a different domain for the same task (paper). For example, if you have a neural net for scene recognition trained on real images of bedrooms, you could reuse the same architecture to recognize drawings of bedrooms. The last few layers represent abstractions like “bed” or “lamp”, which apply to drawings just as well as to real images, while the first few layers represent textures, which would differ between the two data modalities of real images and drawings. More generally, the last few layers are task-dependent and modality-independent, while the first few layers are the opposite.

Importance weighted autoencoders (Ruslan Salakhutdinov)
The variational autoencoder (VAE) is a popular generative model that constructs an autoencoder out of a generative network (encoder) and recognition network (decoder). It then trains these networks to optimize a variational approximation of the posterior distribution by maximizing a lower bound on the log likelihood. IWAE is a variation that tightens the variational lower bound by relaxing the assumptions about the form of the posterior distribution . While the VAE maximizes a lower bound based on a single sample from the recognition distribution, the IWAE lower bound uses a weighted average over several samples. Applying importance weighting over several samples avoids the failure mode where the VAE objective penalizes models that produce even a few samples through the recognition network that don’t fit the posterior from the generative network, and taking several samples allows for better approximation of the posterior and thus a tighter lower bound.(The IWAE paper also gives a more intuitive introduction to VAE than the original paper, in my opinion.)
Variations on RNNs (Yoshua Bengio)
This talk mentioned a few recurrent neural network (RNN) models that were unfamiliar to me. Variational RNNs introduce some elements of variational autoencoders into RNNs by adding latent variables (z) into the top hidden layer (paper). The RNN internal structure is entirely deterministic besides the output probability model, so it can be helpful to inject a higher-level source of noise to model highly structured data (e.g. speech). This was further extended with multiresolution RNNs, which are variational and hierarchical (paper). Another interesting model is real-time recurrent learning, a more biologically plausible alternative to backpropagation through time, where gradients are computed in an online feedforward manner without revisiting past history backwards. The originally proposed version involves a fairly inefficient exact computation of parameter gradients, while a more efficient recent approach approximates the forward gradient instead (paper).
Some other talks I really liked but ran out of steam to write about: Joelle Pineau’s intro to reinforcement learning, Pieter Abbeel on deep reinforcement learning, Shakir Mohamed on deep generative models, Surya Ganguli on neuroscience and deep learning.


Clopen AI: Openness in different aspects of AI development

There has been a lot of discussion about the appropriate level of openness in AI research in the past year – the OpenAI announcement, the blog post Should AI Be Open?, a response to the latter, and Nick Bostrom’s thorough paper Strategic Implications of Openness in AI development.
There is disagreement on this question within the AI safety community as well as outside it. Many people are justifiably afraid of concentrating power to create AGI and determine its values in the hands of one company or organization. Many others are concerned about the information hazards of open-sourcing AGI and the resulting potential for misuse. In this post, I argue that some sort of compromise between openness and secrecy will be necessary, as both extremes of complete secrecy and complete openness seem really bad. The good news is that there isn’t a single axis of openness vs secrecy – we can make separate judgment calls for different aspects of AGI development, and develop a set of guidelines.

Information about AI development can be roughly divided into two categories – technical and strategic. Technical information includes research papers, data, source code (for the algorithm, objective function), etc. Strategic information includes goals, forecasts and timelines, the composition of ethics boards, etc. Bostrom argues that openness about strategic information is likely beneficial both in terms of short- and long-term impact, while openness about technical information is good on the short-term, but can be bad on the long-term due to increasing the race condition. We need to further consider the tradeoffs of releasing different kinds of technical information.
Sharing papers and data is both more essential for the research process and less potentially dangerous than sharing code, since it is hard to reconstruct the code from that information alone. For example, it can be difficult to reproduce the results of a neural network algorithm based on the research paper, given the difficulty of tuning the hyperparameters and differences between computational architectures.
Releasing all the code required to run an AGI into the world, especially before it’s been extensively debugged, tested, and safeguarded against bad actors, would be extremely unsafe. Anyone with enough computational power could run the code, and it would be difficult to shut down the program or prevent it from copying itself all over the Internet.
However, releasing none of the source code is also a bad idea. It would currently be impractical, given the strong incentives for AI researchers to share at least part of the code for recognition and replicability. It would also be suboptimal, since sharing some parts of the code is likely to contribute to safety. For example, it would make sense to open-source the objective function code without the optimization code, which would reveal what is being optimized for but not how. This could make it possible to verify whether the objective is sufficiently representative of society’s values – the part of the system that would be the most understandable and important to the public anyway.
It is rather difficult to verify to what extent a company or organization is sharing their technical information on AI development, and enforce either complete openness or secrecy. There is not much downside to specifying guidelines for what is expected to be shared and what isn’t. Developing a joint set of openness guidelines on the short and long term would be a worthwhile endeavor for the leading AI companies today.
(Cross-posted to the FLI blog and Approximately Correct. Thanks to Jelena Luketina and Janos Kramar for their detailed feedback on this post!)


New AI safety research agenda from Google Brain

Google Brain just released an inspiring research agenda, Concrete Problems in AI Safety, co-authored by researchers from OpenAI, Berkeley and Stanford. This document is a milestone in setting concrete research objectives for keeping reinforcement learning agents and other AI systems robust and beneficial. The problems studied are relevant both to near-term and long-term AI safety, from cleaning robots to higher-stakes applications. The paper takes an empirical focus on avoiding accidents as modern machine learning systems become more and more autonomous and powerful.
Reinforcement learning is currently the most promising framework for building artificial agents – it is thus especially important to develop safety guidelines for this subfield of AI. The research agenda describes a comprehensive (though likely non-exhaustive) set of safety problems, corresponding to where things can go wrong when building AI systems:

Misspecification of the objective function by the human designer. Two common pitfalls when designing objective functions are negative side-effects and reward hacking (also known as wireheading), which are likely to happen by default unless we figure out how to guard against them. One of the key challenges is specifying what it means for an agent to have a low impact on the environment while achieving its objectives effectively.

Extrapolation from limited information about the objective function. Even with a correct objective function, human supervision is likely to be costly, which calls for scalable oversight of the artificial agent.

Extrapolation from limited training data or using an inadequate model. We need to develop safe exploration strategies that avoid irreversibly bad outcomes, and build models that are robust to distributional shift – able to fail gracefully in situations that are far outside the training data distribution.

The AI research community is increasingly focusing on AI safety in recent years, and Google Brain’s agenda is part of this trend. It follows on the heels of the Safely Interruptible Agents paper from Google DeepMind and the Future of Humanity Institute, which investigates how to avoid unintended consequences from interrupting or shutting down reinforcement learning agents. We at FLI are super excited that industry research labs at Google and OpenAI are spearheading and fostering collaboration on AI safety research, and look forward to the outcomes of this work.
(Cross-posted from the FLI blog.)


Using humility to counteract shame

“Pride is not the opposite of shame, but its source. True humility is the only antidote to shame.”
Uncle Iroh, “Avatar: The Last Airbender”

Shame is one of the trickiest emotions to deal with. It is difficult to think about, not to mention discuss with others, and gives rise to insidious ugh fields and negative spirals. Shame often underlies other negative emotions without making itself apparent – anxiety or anger at yourself can be caused by unacknowledged shame about the possibility of failure. It can stack on top of other emotions – e.g. you start out feeling upset with someone, and end up being ashamed of yourself for feeling upset, and maybe even ashamed of feeling ashamed if meta-shame is your cup of tea. The most useful approach I have found against shame is invoking humility.

What is humility, anyway? It is often defined as a low view of your own importance, and tends to be conflated with modesty. Another common definition that I find more useful is acceptance of your own flaws and shortcomings. This is more compatible with confidence, and helpful irrespective of your level of importance or comparison to other people. What humility feels like to me on a system 1 level is a sense of compassion and warmth towards yourself while fully aware of your imperfections (while focusing on imperfections without compassion can lead to beating yourself up). According to LessWrong, “to be humble is to take specific actions in anticipation of your own errors”, which seems more like a possible consequence of being humble than a definition.
Humility is a powerful tool for psychological well-being and instrumental rationality that is more broadly applicable than just the ability to anticipate errors by seeing your limitations more clearly. I can summon humility when I feel anxious about too many upcoming deadlines, or angry at myself for being stuck on a rock climbing route, or embarrassed about forgetting some basic fact in my field that I am surely expected to know by the 5th year of grad school.
While humility comes naturally to some people, others might find it useful to explicitly build an identity as a humble person. How can you invoke this mindset? One way is through negative visualization or pre-hindsight, considering how your plans could fail, which can be time-consuming and usually requires system 2. A faster and less effortful way is to is to imagine a person, real or fictional, who you consider to be humble. I often bring to mind my grandfather, or Uncle Iroh from the Avatar series, sometimes literally repeating the above quote in my head, sort of like an affirmation. I don’t actually agree that humility is the only antidote to shame, but it does seem to be one of the most effective.
(Cross-posted to LessWrong. Thanks to Janos Kramar for his feedback on this post.)

Using humility to counteract shame


Introductory resources on AI safety research

[See AI Safety Resources for the most recent version of this list.]
Reading list to get up to speed on the main ideas in the field of long-term AI safety. The resources are selected for relevance and/or brevity, and the list is not meant to be comprehensive. [Updated on 19 October 2017.]
Motivation
For a popular audience:
Sutskever and Amodei, 2017. Wall Street Journal: Protecting Against AI’s Existential Threat
Cade Metz, 2017. New York Times: Teaching A.I. Systems to Behave Themselves
FLI. AI risk background and FAQ. At the bottom of the background page, there is a more extensive list of resources on AI safety.
Tim Urban, 2015. Wait But Why: The AI Revolution. An accessible introduction to AI risk forecasts and arguments (with cute hand-drawn diagrams, and a few corrections from Luke Muehlhauser).
OpenPhil, 2015. Potential risks from advanced artificial intelligence. An overview of AI risks and timelines, possible interventions, and current actors in this space.

For a more technical audience:
Stuart Russell:

The long-term future of AI (longer version), 2015. A video of Russell’s classic talk, discussing why it makes sense for AI researchers to think about AI safety, and going over various misconceptions about the issues.
Concerns of an AI pioneer, 2015. An interview with Russell on the importance of provably aligning AI with human values, and the challenges of value alignment research.
On Myths and Moonshine, 2014. Russell’s response to the “Myth of AI” question on Edge.org, which draws an analogy between AI research and nuclear research, and points out some dangers of optimizing a misspecified utility function.

Scott Alexander, 2015. No time like the present for AI safety work. An overview of long-term AI safety challenges, e.g. preventing wireheading and formalizing ethics.
Victoria Krakovna, 2015. AI risk without an intelligence explosion. An overview of long-term AI risks besides the (overemphasized) intelligence explosion / hard takeoff scenario, arguing why intelligence explosion skeptics should still think about AI safety.
Stuart Armstrong, 2014. Smarter Than Us: The Rise Of Machine Intelligence. A short ebook discussing potential promises and challenges presented by advanced AI, and the interdisciplinary problems that need to be solved on the way there.
Technical overviews
Soares and Fallenstein, 2017. Aligning Superintelligence with Human Interests: A Technical Research Agenda
Amodei, Olah, et al, 2016. Concrete Problems in AI safety. Research agenda focusing on accident risks that apply to current ML systems as well as more advanced future AI systems.
Jessica Taylor et al, 2016. Alignment for Advanced Machine Learning Systems
FLI, 2015. A survey of research priorities for robust and beneficial AI
Jacob Steinhardt, 2015. Long-Term and Short-Term Challenges to Ensuring the Safety of AI Systems. A taxonomy of AI safety issues that require ordinary vs extraordinary engineering to address.
Nate Soares, 2015. Safety engineering, target selection, and alignment theory. Identifies and motivates three major areas of AI safety research.
Nick Bostrom, 2014. Superintelligence: Paths, Dangers, Strategies. A seminal book outlining long-term AI risk considerations.
Steve Omohundro, 2007. The basic AI drives. A classic paper arguing that sufficiently advanced AI systems are likely to develop drives such as self-preservation and resource acquisition independently of their assigned objectives.
Technical work
Value learning:
Jaime Fisac et al, 2017. Pragmatic-Pedagogic Value Alignment. A cognitive science approach to the cooperative inverse reinforcement learning problem.
Smitha Milli et al. Should robots be obedient? Obedience to humans may sound like a great thing, but blind obedience can get in the way of learning human preferences.
William Saunders et al, 2017. Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. (blog post)
Amin, Jiang, and Singh, 2017. Repeated Inverse Reinforcement Learning. Separates the reward function into a task-specific component and an intrinsic component. In a sequence of task, the agent learns the intrinsic component while trying to avoid surprising the human.
Armstrong and Leike, 2016. Towards Interactive Inverse Reinforcement Learning. The agent gathers information about the reward function through interaction with the environment, while at the same time maximizing this reward function, balancing the incentive to learn with the incentive to bias.
Dylan Hadfield-Menell et al, 2016. Cooperative inverse reinforcement learning. Defines value learning as a cooperative game where the human tries to teach the agent about their reward function, rather than giving optimal demonstrations like in standard IRL.
Owain Evans et al, 2016. Learning the Preferences of Ignorant, Inconsistent Agents.
Reward gaming / wireheading:
Tom Everitt et al, 2017. Reinforcement learning with a corrupted reward channel. A formalization of the reward misspecification problem in terms of true and corrupt reward, a proof that RL agents cannot overcome reward corruption, and a framework for giving the agent extra information to overcome reward corruption. (blog post)
Amodei and Clark, 2016. Faulty Reward Functions in the Wild. An example of reward function gaming in a boat racing game, where the agent gets a higher score by going in circles and hitting the same targets than by actually playing the game.
Everitt and Hutter, 2016. Avoiding Wireheading with Value Reinforcement Learning. An alternative to RL that reduces the incentive to wirehead.
Laurent Orseau, 2015. Wireheading. An investigation into how different types of artificial agents respond to opportunities to wirehead (unintended shortcuts to maximize their objective function).
Interruptibility / corrigibility:
Dylan Hadfield-Menell et al. The Off-Switch Game. This paper studies the interruptibility problem as a game between human and robot, and investigates which incentives the robot could have to allow itself to be switched off.
El Mahdi El Mhamdi et al, 2017. Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning.
Orseau and Armstrong, 2016. Safely interruptible agents. Provides a formal definition of safe interruptibility and shows that off-policy RL agents are more interruptible than on-policy agents. (blog post)
Nate Soares et al, 2015. Corrigibility. Designing AI systems without incentives to resist corrective modifications by their creators.
Scalable oversight:
Christiano, Leike et al, 2017. Deep reinforcement learning from human preferences. Communicating complex goals to AI systems using human feedback (comparing pairs of agent trajectory segments).
David Abel et al. Agent-Agnostic Human-in-the-Loop Reinforcement Learning.
Other:
Armstrong and Levinstein, 2017. Low Impact Artificial Intelligences. An intractable but enlightening definition of low impact for AI systems.
Babcock, Kramar and Yampolskiy, 2017. Guidelines for Artificial Intelligence Containment.
Scott Garrabrant et al, 2016. Logical Induction. A computable algorithm for the logical induction problem.
Note: I did not include literature on less neglected areas of the field like safe exploration, distributional shift, adversarial examples, or interpretability (see e.g. Concrete Problems or the CHAI bibliography for extensive references on these topics).
Collections of technical works
CHAI bibliography
MIRI publications
FHI publications
FLI grantee publications (scroll down)
Paul Christiano. AI control. A blog on designing safe, efficient AI systems (approval-directed agents, aligned reinforcement learning agents, etc).
If there are any resources missing from this list that you think are a must-read, please let me know! If you want to go into AI safety research, check out these guidelines and the AI Safety Syllabus.
(Thanks to Ben Sancetta, Taymon Beal and Janos Kramar for their feedback on this post.)


To contribute to AI safety, consider doing AI research

Among those concerned about risks from advanced AI, I’ve encountered people who would be interested in a career in AI research, but are worried that doing so would speed up AI capability relative to safety. I think it is a mistake for AI safety proponents to avoid going into the field for this reason (better reasons include being well-positioned to do AI safety work, e.g. at MIRI or FHI). This mistake contributed to me choosing statistics rather than computer science for my PhD, which I have some regrets about, though luckily there is enough overlap between the two fields that I can work on machine learning anyway.
I think the value of having more AI experts who are worried about AI safety is far higher than the downside of adding a few drops to the ocean of people trying to advance AI. Here are several reasons for this:

Concerned researchers can inform and influence their colleagues, especially if they are outspoken about their views.
Studying and working on AI brings understanding of the current challenges and breakthroughs in the field, which can usefully inform AI safety work (e.g. wireheading in reinforcement learning agents).
Opportunities to work on AI safety are beginning to spring up within academia and industry, e.g. through FLI grants. In the next few years, it will be possible to do an AI-safety-focused PhD or postdoc in computer science, which would hit two birds with one stone.

To elaborate on #1, one of the prevailing arguments against taking long-term AI safety seriously is that not enough experts in the AI field are worried. Several prominent researchers have commented on the potential risks (Stuart Russell, Bart Selman, Murray Shanahan, Shane Legg, and others), and more are concerned but keep quiet for reputational reasons. An accomplished, strategically outspoken and/or well-connected expert can make a big difference in the attitude distribution in the AI field and the level of familiarity with the actual concerns (which are not about malevolence, sentience, or marching robot armies). Having more informed skeptics who have maybe even read Superintelligence, and fewer uninformed skeptics who think AI safety proponents are afraid of Terminators, would produce much needed direct and productive discussion on these issues. As the proportion of informed and concerned researchers in the field approaches critical mass, the reputational consequences for speaking up will decrease.
A year after FLI’s Puerto Rico conference, the subject of long-term AI safety is no longer taboo among AI researchers, but remains rather controversial. Addressing AI risk on the long term will require safety work to be a significant part of the field, and close collaboration between those working on safety and capability of advanced AI. Stuart Russell makes the apt analogy that “just as nuclear fusion researchers consider the problem of containment of fusion reactions as one of the primary problems of their field, issues of control and safety will become central to AI as the field matures”. If more people who are already concerned about AI safety join the field, we can make this happen faster, and help wisdom win the race with capability.
(Cross-posted to LessWrong. Thanks to Janos Kramar for his help with editing this post.)


2015-16 New Year review

2015 progress
Research:

Finished paper on the Selective Bayesian Forest Classifier algorithm
Made an R package for SBFC (beta)
Worked at Google on unsupervised learning for the Knowledge Graph with Moshe Looks during the summer (paper)
Joined the HIPS research group at Harvard CS and started working with the awesome Finale Doshi-Velez
Ratio of coding time to writing time was too high overall

FLI:

Co-organized two meetings to brainstorm biotechnology risks
Co-organized two Machine Learning Safety meetings
Gave a talk at the Shaping Humanity’s Trajectory workshop at EA Global
Helped organize NIPS symposium on societal impacts of AI

Rationality / effectiveness:

Extensive use of FollowUpThen for sending reminders to future selves
Mapped out my personal bottlenecks
Sleep:

Tracked insomnia (26% of nights) and sleep time (average 1:30am, stayed up past 1am on 31% of nights)
Started working on sleep hygiene
Stopped using melatonin (found it ineffective)

Random cool things I did:

Improv class
Aerial silks class
Climbed out of a glacial abyss (moulin)
Placed second at Toastmasters area speech contest

2015 prediction outcomes
Out of the 17 predictions I made a year ago, 5 were true, and the rest were false.

Submit the SBFC paper for publication (95%)
Submit another paper besides SBFC (40%)
Present SBFC results at a conference (JSM, ICML or NIPS) (40%) – presented at a workshop (NESS)
Get a new external fellowship to replace my expiring NSERC fellowship (50%)
Skim at least 20 research papers in machine learning (70%) – probably a lot more
Write at least 12 blog posts (70%) – wrote 9 posts
Climb a 5.12 without rope cheating (50%) – no longer endorsed at this level
Lead climb a 5.11a (50%) – no longer endorsed at this level
Do 10 pullups in a row (60%) – no longer endorsed at this level
Meditate at least 150 times (80%) – 206 times
Record at least 150 new thoughts (70%) – recorded 62, no longer endorsed at this level
Make at least 100 Anki cards by the end of the year (70%)
Read at least 10 books (60%) – read 4 books, no longer endorsed at this level
Attend Burning Man (90%)
Boston will have a second rationalist house by the end of the year (30%)
FLI will hire a full-time project manager or administrator (80%) – no, but we now have a full time website editor…
FLI will start a project on biotech safety (70%) – had some meetings, but no concrete action plan yet

Calibration:

low predictions, 30-60%: 0/8 = 0% (super overconfident)
high predictions, 70-95%: 5/9 = 56% (overconfident)

(Yikes! Worse than last year…)
Conclusions:

I forgot about most of these goals after a few months – will need a recurring reminder for next year.
All 3 physical goals ended up disendorsed – I think I set those way too high. My climbing habits got disrupted by moving to California in summer and a hand injury, so I’m still trying to return to my spring 2014 skill level.

2016 goals and predictions
Given the overconfidence of last year’s predictions, toning it down for next year.
Resolutions:

Finish PhD thesis (70%)
Write at least 12 blog posts (40%)
Meditate at least 200 days (50%)
Exercise at least 200 days (50%)
Do at least 5 pullups in a row (40%)
Record at least 50 new thoughts (50%)
Stay up at most 20% of the nights (40%)
Do at least 10 pomodoros per week on average (50%)

Predictions:

At least one paper accepted for publication (70%)
I will get at least one fellowship (40%)
Insomnia at most 20% of nights (20%)
FLI will co-organize at least 3 AI safety workshops (50%)

2015-16 New Year review


Highlights and impressions from NIPS conference on machine learning

This year’s NIPS was an epicenter of the current enthusiasm about AI and deep learning – there was a visceral sense of how quickly the field of machine learning is progressing, and two new AI startups were announced. Attendance has almost doubled compared to the 2014 conference (I hope they make it multi-track next year), and several popular workshops were standing room only. Given that there were only 400 accepted papers and almost 4000 people attending, most people were there to learn and socialize. The conference was a socially intense experience that reminded me a bit of Burning Man – the overall sense of excitement, the high density of spontaneous interesting conversations, the number of parallel events at any given time, and of course the accumulating exhaustion.
Some interesting talks and posters
Sergey Levine’s robotics demo at the crowded Deep Reinforcement Learning workshop (we showed up half an hour early to claim spots on the floor). This was one of the talks that gave me a sense of fast progress in the field. The presentation started with videos from this summer’s DARPA robotics challenge, where the robots kept falling down while trying to walk or open a door. Levine proceeded to outline his recent work on guided policy search, alternating between trajectory optimization and supervised training of the neural network, and granularizing complex tasks. He showed demos of robots successfully performing various high-dexterity tasks, like opening a door, screwing on a bottle cap, or putting a coat hanger on a rack. Impressive!

Generative image models using a pyramid of adversarial networks by Denton & Chintala. Generating realistic-looking images using one neural net as a generator and another as an evaluator – the generator tries to fool the evaluator by making the image indistinguishable from a real one, while the evaluator tries to tell real and generated images apart. Starting from a coarse image, successively finer images are generated using the adversarial networks from the coarser images at the previous level of the pyramid. The resulting images were mistaken for real images 40% of the time in the experiment, and around 80% of them looked realistic to me when staring at the poster.
Path-SGD by Salakhutdinov et al, a scale-invariant version of the stochastic gradient descent algorithm. Standard SGD uses the L2 norm in as the measure of distance in the parameter space, and rescaling the weights can have large effects on optimization speed. Path-SGD instead regularizes the maximum norm of incoming weights into any unit, minimizing the max-norm over all rescalings of the weights. The resulting norm (called a “path regularizer”) is shown to be invariant to weight rescaling. Overall a principled approach with good empirical results.
End-to-end memory networks by Sukhbaatar et al (video), an extension of memory networks – neural networks that learn to read and write to a memory component. Unlike traditional memory networks, the end-to-end version eliminates the need for supervision at each layer. This makes the method applicable to a wider variety of domains – it is competitive both with memory networks for question answering and with LSTMs for language modeling. It was fun to see the model perform basic inductive reasoning about locations, colors and sizes of objects.
Neural GPUs (video), Deep visual analogy-making (video), On-the-job learning, and many others.
Algorithms Among Us symposium (videos)
A highlight of the conference was the Algorithms Among Us symposium on the societal impacts of machine learning, which I helped organize along with others from FLI. The symposium consisted of 3 panels and accompanying talks – on near-term AI impacts, timelines to general AI, and research priorities for beneficial AI. The symposium organizers (Adrian Weller, Michael Osborne and Murray Shanahan) gathered an impressive array of AI luminaries with a variety of views on the subject, including Cynthia Dwork from Microsoft, Yann LeCun from Facebook, Andrew Ng from Baidu, and Shane Legg from DeepMind. All three panel topics generated lively debate among the participants.

3 panels at #AlgorithmsAmongUs symposium #NIPS2015 pic.twitter.com/X1KNAwwkWW
— Victoria Krakovna (@vkrakovna) December 15, 2015

Andrew Ng took his famous statement that “worrying about general AI is like worrying about overpopulation on Mars” to the next level, namely “overpopulation on Alpha Centauri” (is Mars too realistic these days?). His main argument was that even superforecasters can’t predict anything 5 years into the future, so any predictions on longer time horizons are useless. This seemed like an instance of the all-too-common belief that “we don’t know, therefore we are safe”. As Murray pointed out, having complete uncertainty past a 5-year horizon means that you can’t rule out reaching general AI in 20 years either. Encouragingly, Ng endorsed long-term AI safety research, saying that it’s not his cup of tea but someone should be working on it.
With regards to roadmapping the remaining milestones to general AI, Yann LeCun gave an apt analogy of traveling through mountains in the fog – there are some you can see, and an unknown number hiding in the fog. He also argued that advanced AI is unlikely to be human-like, and cautioned against anthropomorphizing it.
In the research priorities panel, Shane Legg gave some specific recommendations – goal system stability, interruptibility, sandboxing / containment, and formalization of various thought experiments (e.g. in Superintelligence). He pointed out that AI safety is both overblown and underemphasized – while the risks from advanced AI are not imminent the way they are usually portrayed in the media, more thought and resources need to be devoted to the challenging research problems involved.
One question that came up during the symposium is the importance of interpretability for AI systems, which is actually the topic of my current research project. There was some disagreement about the tradeoff between effectiveness and interpretability. LeCun thought that the main advantage of interpretability is increased robustness, and improvements to transfer learning should produce that anyway, without decreases in effectiveness. Percy Liang argued that transparency is needed to explain to the rest of the world what machine learning systems are doing, which is increasingly important in many applications. LeCun also pointed out that machine learning systems that are usually considered transparent, such as decision trees, aren’t necessarily so. There was also disagreement about what interpretability means in the first place – as Cynthia Dwork said, we need a clearer definition before making any conclusions. It seems that more work is needed both on defining interpretability and on figuring out how to achieve it without sacrificing effectiveness.
Overall, the symposium was super interesting and gave a lot of food for thought (here’s a more detailed summary by Ariel from FLI). Thanks to Adrian, Michael and Murray for their hard work in putting it together.
AI startups
It was exciting to see two new AI startups announced at NIPS – OpenAI, led by Ilya Sutskever and backed by Musk, Altman and others, and Geometric Intelligence, led by Zoubin Ghahramani and Gary Marcus.
OpenAI is a non-profit with a mission to democratize AI research and keep it beneficial for humanity, and a whopping $1Bn in funding pledged. They believe that it’s safer to have AI breakthroughs happening in a non-profit, unaffected by financial interests, rather than monopolized by for-profit corporations. The intent to open-source the research seems clearly good in the short and medium term, but raises some concerns in the long run when getting closer to general AI. As an OpenAI researcher emphasized in an interview, “we are not obligated to share everything – in that sense the name of the company is a misnomer”, and decisions to open-source the research would in fact be made on a case-by-case basis.
While OpenAI plans to focus on deep learning in their first few years, Geometric Intelligence is developing an alternative approach to deep learning that can learn more effectively from less data. Gary Marcus argues that we need to learn more from how human minds acquire knowledge in order to build advanced AI (an inspiration for the venture was observing his toddler learn about the world). I’m looking forward to what comes out of the variety of approaches taken by these new companies and other research teams.
(Cross-posted on the FLI blog. Thanks to Janos Kramar for his help with editing this post.)


Risks from general artificial intelligence without an intelligence explosion

“An ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”
– Computer scientist I. J. Good, 1965
Artificial intelligence systems we have today can be referred to as narrow AI – they perform well at specific tasks, like playing chess or Jeopardy, and some classes of problems like Atari games. Many experts predict that general AI, which would be able to perform most tasks humans can, will be developed later this century, with median estimates around 2050. When people talk about long term existential risk from the development of general AI, they commonly refer to the intelligence explosion (IE) scenario. AI risk skeptics often argue against AI safety concerns along the lines of “Intelligence explosion sounds like science-fiction and seems really unlikely, therefore there’s not much to worry about”. It’s unfortunate when AI safety concerns are rounded down to worries about IE. Unlike I. J. Good, I do not consider this scenario inevitable (though relatively likely), and I would expect general AI to present an existential risk even if I knew for sure that intelligence explosion were impossible.

Here are some dangerous aspects of developing general AI, besides the IE scenario:

Human incentives. Researchers, companies and governments have professional and economic incentives to build AI that is as powerful as possible, as quickly as possible. There is no particular reason to think that humans are the pinnacle of intelligence – if we create a system without our biological constraints, with more computing power, memory, and speed, it could become more intelligent than us in important ways. The incentives are to continue improving AI systems until they hit physical limits on intelligence, and those limitations (if they exist at all) are likely to be above human intelligence in many respects.
Convergent instrumental goals. Sufficiently advanced AI systems would by default develop drives like self-preservation, resource acquisition, and preservation of their objective functions, independent of their objective function or design. This was outlined in Omohundro’s paper and more concretely formalized in a recent MIRI paper. Humans routinely destroy animal habitats to acquire natural resources, and an AI system with any goal could always use more data centers or computing clusters.
Unintended consequences. As in the stories of Sorcerer’s Apprentice and King Midas, you get what you asked for, but not what you wanted. This already happens with narrow AI, like in the frequently cited example from the Bird & Layzell paper: a genetic algorithm was supposed to design an oscillator using a configurable circuit board, and instead designed a makeshift radio that used signal from neighboring computers to produce the requisite oscillating pattern. Unintended consequences produced by a general AI, more opaque and more powerful than a narrow AI, would likely be far worse.
Value learning is hard. Specifying common sense and ethics in computer code is no easy feat. As argued by Stuart Russell, given a misspecified value function that omits variables that turn out to be important to humans, an optimization process is likely to set these unconstrained variables to extreme values. Think of what would happen if you asked a self-driving car to get you to the airport as fast as possible, without assigning value to obeying speed limits or avoiding pedestrians. While researchers would have incentives to build in some level of common sense and understanding of human concepts that is needed for commercial applications like household robots, that might not be enough for general AI.
Value learning is insufficient. Even an AI system with perfect understanding of human values and goals would not necessarily adopt them. Humans understand the “goals” of the evolutionary process that generated us, but don’t internalize them – in fact, we often “wirehead” our evolutionary reward signals, e.g. by eating sugar.
Containment is hard. A general AI system with access to the internet would be able to hack thousands of computers and copy itself onto them, thus becoming difficult or impossible to shut down – this is a serious problem even with present-day computer viruses. When developing an AI system in the vicinity of general intelligence, it would be important to keep it cut off from the internet. Large scale AI systems are likely to be run on a computing cluster or on the cloud, rather than on a single machine, which makes isolation from the internet more difficult. Containment measures would likely pose sufficient inconvenience that many researchers would be tempted to skip them.

Some believe that if intelligence explosion does not occur, AI progress will occur slowly enough that humans can stay in control. Given that human institutions like academia or governments are fairly slow to respond to change, they may not be able to keep up with an AI that attains human-level or superhuman intelligence over months or even years. Humans are not famous for their ability to solve coordination problems. Even if we retain control over AI’s rate of improvement, it would be easy for bad actors or zealous researchers to let it go too far – as Geoff Hinton recently put it, “the prospect of discovery is too sweet”.
As a machine learning researcher, I care about whether my field will have a positive impact on humanity in the long term. The challenges of AI safety are numerous and complex (for a more technical and thorough exposition, see Jacob Steinhardt’s essay), and cannot be rounded off to a single scenario. I look forward to a time when disagreements about AI safety no longer derail into debates about IE, and instead focus on other relevant issues we need to figure out.
(Cross-posted on the FLI blog. Thanks to Janos Kramar for his help with editing this post.)


Nomadism and Burning Man

The past few months had been quite nomadic even by my standards. Visiting the MIRI fellows program, EA Global, Alaska camping (in 4 different parks), CFAR alumni reunion, and finally Burning Man. It was an exciting social time – I had many great conversations and coordinated with various people. It was also tiring to keep up with all the schedule changes, packing and unpacking, and my habits fell through the cracks (meditation, exercise and the like). It is a relief to be back at Citadel with all my stuff in one place, in a stable work and social environment. It almost feels like I’d never left.
It seemed appropriate to conclude the wanderings with Burning Man, which was the most like an actual vacation for me. This year was a good combination of spontaneity and scheduling, adventures and conversations.

Janos and I practiced the beginner mindset at various circus arts. In the long wait on the road into BRC, it turned out that the car in front of us had some hula hoops, buugengs and fans, and people who knew how to use them and were willing to teach. A buugeng is an S-shaped wooden object that can be twisted and thrown around – learning to throw it over my shoulder and catch it involved a lot of running around and hitting my fingers, but I got it eventually. We discovered that our camp was right next to the Hellfire Society, which ran fire spinning classes. We were late for the actual instruction, but got to do a 30-second demo with a real fire staff (turns out, even the most basic clockwise spin looks cool with real fire), and watch Michael Vassar spin fire poi like a pro. We also went to an aerial silks workshop, which inspired at least one friend to continue learning this in the default world.
We got to fight in the Thunderdome! This consists of whacking each other with foam swords while hanging from the ceiling in harnesses. The fighter who is judged to be the most aggressive wins, though they make a bigger deal out of the show than the outcome. Part of the challenge is retaining contact with the other person (or their sword) while whacking them with your sword as much as possible. We stood in line for an hour watching other people fight, despite our attempts to bribe the leather-clad staff with a bottle of alcohol to skip the line (they refused to accept anything other than their customary bribe of good whiskey). The deafening metal music and the shouts of the numerous spectators built up a lot of adrenaline. I promptly forgot all the advice the staff gave me about positions and strategies, and just had fun getting my ass kicked by Sam. Janos fought a guy who was a head taller than him and skilled in martial arts, who he managed to disarm twice. We left the place excitedly tired, and spent the rest of the evening in a relaxing cuddle puddle.
Our playa adventures were interspersed with various conversations and reflections:

A (surprisingly analytical) talk about shame at Mystic camp. The main insight was that a lot of what sustains addictions is the shame around them, and the shame itself is addictive. One prediction made by this theory is that signaling endorsement for addictive activities (e.g. eating junk food on a nice plate at a candlelit table) reduces their addictiveness a lot. It might be useful to install a trigger-action plan like “notice I am ashamed of something I do regularly -> try self-signaling endorsement for the activity”.
A discussion about common sources of motivation for doing things. I’m often motivated by importance or indignation about the status quo, while puzzle-solving motivation is something I used to engage more in my math contest days, and could use more of now. There is also a kind of motivation to do procedural things with my hands, like helping to build camp, which could be called tactile motivation, though that doesn’t sound quite right.
Learning about an underappreciated physics phenomenon called counterflow heat exchange, which allows two containers of fluid to go almost all the way towards swapping temperatures. (h/t Danielle Fong)

Burning Man is a good place to practice spontaneity, going along with whatever comes up. This year I tried to retain this frame of mind even while going to scheduled events. The go-to phrase that I kept repeating in my head was “roll with it”. This was actually quite effective at eliminating fear of missing out, which is usually pervasive in a place where a hundred events are going on at any given time.
This is also the only time when I actually manage to “unplug” for a whole week. It doesn’t seem to work during other vacations – for example, in Alaska I kept looking for wifi during rest stops in towns. Burning Man, on the other hand, was both sufficiently stimulating and sufficiently remote that I experienced no desire to check email or other internet things, even though there is actually a wifi camp on the playa. Occasionally reminding my system 1 that the sky does not fall if I ignore my email for a week is pretty valuable. I am now trying to sustain the spontaneity and email-ignoring mindsets in the default world to some extent.

Nomadism and Burning Man


Systems I have tried: an overview

I have used various organization and productivity systems in the past few years – this is an overview of what worked and what didn’t.
Main systems I currently use:

Follow Up Then: Sends an email to a future self, with the date and time specified in the email address, e.g. fri6pm@fut.io. I use it for delaying tasks, recurring reminders, and following up on email threads. This reduces clutter in my todo list, calendar and inbox, and frees my working memory. Lately, I noticed myself remembering a thing shortly before receiving a follow up about it – probably due to the same mechanism that sometimes wakes me up a few minutes before the morning alarm.
Complice: Daily to-do list organized according to goals, with archives and regular reviews. Helpful for specifying the next action to take at a given time, and for tracking progress on individual goals. Downside: I sometimes hesitate to enter tasks into the list, because entered tasks cannot be erased, and leaving a task unfinished is aversive, so often end up entering tasks after they are done instead.
Workflowy: Nested list structure – searchable, with collapsible and sharable sublists. I keep my ongoing todo list (in GTD form) and most of my notes here. Downside: doesn’t work for goal factoring, since it only supports tree structures.
Google Calendar: Self-explanatory. I have recently started adding tentative meeting slots, indicated by a question mark, e.g. “dinner with Janos?”. This has been helpful for keeping track of which time slots I’ve offered to someone. I also added a calendar that shows Facebook events that I’ve been invited to, which is handy.
42 Goals: Goal tracking with summary graphs and cute symbols. I use this for tracking habits (like exercise and meditation) and other random things (like insomnia occurrences). The graphs are useful – this is how I know that I have the most insomnia on Mondays! Downsides: doesn’t allow non-binary categories, and the phone app is so unreliable that I never use it – if you know good alternative tracking systems, let me know!

Systems I no longer use:

Beeminder: Goal tracking with nice graphs, and goal setting with reminders and financial penalties in case of failure. I liked the graphs and reminders, but the penalties made me feel even more overwhelmed than usual, and sometimes induced suboptimal short-term priorities. I decided to obtain the different benefits separately, setting recurring reminders for habits on Follow Up Then, and using 42 Goals for tracking.
Toggl: Time tracking for activities and tasks, organized by project or goal, with an option for retroactive time entries. I started out using it to track all my time, and though I stopped after about a month due to the excessive overhead of tracking and categorizing short activities, I learned a lot about where my time was going. I used it for about a year after that to track work hours, and eventually stopped because of overhead and redundancy with Complice.
Paper checklist: Checklist for daily habits. Worked well in terms of catching my eye in the morning, but was often forgotten when traveling. It was redundant with 42 Goals, and required double data entry, so I eventually gave up on the paper version.
Coach.me: Habit tracking with reminders, with a pretty good phone app. I found it particularly useful for several-times-a-week habits. It also has built-in habit programs like building up to a certain number of chinups. I mostly stopped using it because I had too many other systems that were redundant with it.
Pomodoros: Setting a timer to focus on a specific task for 25-40 minutes, followed by a break of 5 minutes. I found it unpleasant to be forced to take breaks, developed a habit of ignoring the break signal, and gave up on using pomodoros altogether.

Over the past couple of years, I have become less willing to force myself to do things or overwhelm myself with instructions or data entry overhead, which has led me to reduce the number of systems I use, and to prefer gently guiding systems to strict ones.

Systems I have tried: an overview


Hamming questions and bottlenecks

The CFAR alumni workshop on the first weekend of May was focused on the Hamming question. Mathematician Richard Hamming was known to approach experts from other fields and ask “what are the important problems in your field, and why aren’t you working on them?”. The same question can be applied to personal life: “what are the important problems in your life, and what is stopping you from working on them?”.
Over the course of the weekend, the twelve of us asked this question of ourselves and each other, in many forms and guises: “if Vika isn’t making a major impact on the world in 5 years, what would have stopped her?”, “what are your greatest bottlenecks?”, “how can we actually try?”, etc. The intense focus on mental pain points was interspersed with naps and silly games to let off steam. On the last day, we did a group brainstorm, where everyone who wanted to receive feedback took a turn in the center of the circle, and everyone else speculated on what they thought were the biggest bottlenecks of the person in the center. By this time, we had mostly gotten to know each other, and even the impressions from those who knew me less well were surprisingly accurate. I am very grateful to everyone at the workshop for being so insightful and supportive of each other (and actually caring).

Most of the issues that came up were things I was aware of on some level, but over the course of the workshop it became particularly salient to me how interconnected my problems are and how the gears in the system affect each other. Working memory overload leads to confusion, which reduces confidence. Sleep deprivation reduces working memory and increases anxiety. Anxiety reduces the affordance for exploration and creativity, and increases the frequency of insomnia. Ignoring or neglecting signals from system 1 takes up working memory slots with looping messages from system 1, and increases anxiety. After a few of these circular explanations, I gave up on writing them all down, and made a diagram instead.

A few things jump out at me about this diagram. The highest degree nodes are anxiety and working memory, both of which are difficult to affect directly. The two nodes I have the most influence over are the amount of sleep I get and the degree to which I listen to system 1 signals. I have started experimenting with sleep interventions that I haven’t yet tried, like taking melatonin 4 hours before bedtime, using a weighted blanket, etc. Attunement to system 1 can be improved through meditation, Focusing, belief reporting and such. While I have sporadically meditated for years, I could use more practice at the other techniques, which involve more explicit internal querying than meditation.
Curiously, the graph also appears to have a source and a sink. The source node is my overdeveloped sense of duty and a tendency to assume I should do things or be able to do things causes a lot of downstream issues. It would be impactful to directly hack this and become more selfish, but it appears to be a bit trickier than doing a find-and-replace on my source code, replacing “I have to do X” with “my goals require X”. The sink node has to do with my capacity to allow myself time and mental space for exploration and creativity, which would among other things enable me to do my high-level goals better (e.g. research and organization strategy).
A week after the workshop, I moved to California for my summer internship at Google. The context shift and my new location a few blocks down from the CFAR office will allow me to work on my bottlenecks more systematically. I have wrestled with these for a long time, but now I feel that I have better tools and resources than ever before.

Hamming questions and bottlenecks


Negative visualization, radical acceptance and stoicism

In anxious, frustrating or aversive situations, I find it helpful to visualize the worst case that I fear might happen, and try to accept it. I call this “radical acceptance”, since the imagined worst case is usually an unrealistic scenario that would be extremely unlikely to happen, e.g. “suppose I get absolutely nothing done in the next month”. This is essentially the negative visualization component of stoicism.
There are many benefits to visualizing the worst case:

Feeling better about the present situation by contrast.
Turning attention to the good things that would still be in my life even if everything went wrong in one particular domain.
Weakening anxiety using humor (by imagining an exaggerated “doomsday” scenario).
Being more prepared for failure, and making contingency plans (pre-hindsight).
Helping make more accurate predictions about the future by reducing the “X isn’t allowed to happen” effect (or, as Anna Salamon once put it, “putting X into the realm of the thinkable”).
Reducing the effect of ugh fields / aversions, which thrive on the “X isn’t allowed to happen” flinch.
Weakening unhelpful identities like “person who is always productive” or “person who doesn’t make stupid mistakes”.

Let’s say I have an aversion around meetings with my advisor, because I expect him to be disappointed with my research progress. When I notice myself worrying about the next meeting or finding excuses to postpone it so that I have more time to make progress, I can imagine the worst imaginable outcome a meeting with my advisor could have – perhaps he might yell at me or even decide to expel me from grad school (neither of these have actually happened so far). If the scenario is starting to sound silly, that’s a good sign. I can then imagine how this plays out in great detail, from the disappointed faces and words of the rest of the department to the official letter of dismissal in my hands, and consider what I might do in that case, like applying for industry jobs. While building up these layers of detail in my mind, I breathe deeply, which I associate with meditative acceptance of reality. (I use the word “acceptance” to mean “acknowledgement” rather than “resignation”.)
I am trying to use this technique more often, both in the regular and situational sense. A good default time is my daily meditation practice. I might also set up a trigger-action habit of the form “if I notice myself repeatedly worrying about something, visualize that thing (or an exaggerated version of it) happening, and try to accept it”. Some issues have more natural triggers than others – while worrying tends to call attention to itself, aversions often manifest as a quick flinch away from a thought, so it’s better to find a trigger among the actions that are often caused by an aversion, e.g. procrastination. A trigger for a potentially unhelpful identity could be a thought like “I’m not good at X, but I should be”. A particular issue can simultaneously have associated worries (e.g. “will I be productive enough?”), aversions (e.g. towards working on the project) and identities (“productive person”), so there is likely to be something there that makes a good trigger. Visualizing myself getting nothing done for a month can help with all of these to some degree.
System 1 is good at imagining scary things – why not use this as a tool?
Cross-posted


Future of Life Institute’s recent milestones in AI safety

In January, many months of effort by FLI’s founders and volunteers finally came to fruition – the Puerto Rico conference, open letter and grants program announcement took place in rapid succession. The conference was a resounding success according to many of the people there, who were impressed with the quality of the ideas presented and the way it was organized. There were opportunities for the attendees to engage with each other at different levels of structure, from talks to panels to beach breakout groups. The relaxed Caribbean atmosphere seemed to put everyone at ease, and many candid and cooperative conversations happened between attendees with rather different backgrounds and views.
It was fascinating to observe many of the AI researchers get exposed to various AI safety ideas for the first time. Stuart Russell’s argument that the variables that are not accounted for by the objective function tend to be pushed to extreme values, Nick Bostrom’s presentation on takeoff speeds and singleton/multipolar scenarios, and other key ideas were received quite well. One attending researcher summed it up along these lines: “It is so easy to obsess about the next building block towards general AI, that we often forget to ask ourselves a key question – what happens when we succeed?”.

A week after the conference, the open letter outlining the research priorities went public. The letter and research document were the product of many months of hard work and careful thought by Daniel Dewey, Max Tegmark, Stuart Russell, and others. It was worded in optimistic and positive terms – the most negative word in the whole thing was “pitfalls”. Nevertheless, the media’s sensationalist lens twisted the message into things like “experts pledge to rein in AI research” to “warn of a robot uprising” and “protect mankind from machines”, invariably accompanied by a Terminator image or a Skynet reference. When the grants program was announced soon afterwards, the headlines became “Elon Musk donates…” to “keep killer robots at bay”, “keep AI from turning evil”, you name it. Those media portrayals shared a key misconception of the underlying concerns, that AI has to be “malevolent” to be dangerous, while the most likely problematic scenario in our minds is a misspecified general AI system with beneficial or neutral objectives. While a few reasonable journalists actually bothered to get in touch with FLI and discuss the ideas behind our efforts, most of the media herd stampeded ahead under the alarmist Terminator banner.
The open letter expresses a joint effort by the AI research community to step up to the challenge of advancing AI safety as responsible scientists. My main worry about this publicity angle is that this might be the first major exposure to AI safety concerns for many people, including AI researchers who would understandably feel attacked and misunderstood by the media’s framing of their work. It is really unfortunate to have some researchers turned away from the cause of keeping AI beneficial and safe without even engaging with the actual concerns and arguments.
I am sometimes asked by reporters whether there has been too much emphasis on the superintelligence concerns that is “distracting” from the more immediate AI impacts like the automation of jobs and autonomous weapons. While the media hype is certainly not helpful towards making progress on either the near-term or long-term concerns, there is a pervasive false dichotomy here, as both of these domains are in dire need of more extensive research. The near-term economic and legal issues are already on the horizon, while the design and forecasting of general AI is a complex interdisciplinary research challenge that will likely take decades, so it is of utmost importance to begin the work as soon as possible.
The grants program on AI safety, fueled by Elon Musk’s generous donation, is now well under way, with the initial proposals due March 1. The authors of the best initial proposals will be invited to submit a more detailed full proposal by May 17. I hope that our program will help kickstart the emerging subfield of AI safety, stimulate open discussion of the ideas among the AI experts, and broaden the community of researchers working on these important and difficult questions. Stuart Russell put it well in his talk at the Puerto Rico conference: “Solving this problem should be an intrinsic part of the field, just as containment is a part of fusion research. It isn’t ‘Ethics of AI’, it’s common sense!”.


2014-15 New Year review

2014 progress
If someone told me at the beginning of 2014 that I would co-found an organization to mitigate technological risks to humanity, I might not have believed them. Thanks Max, Meia, Anthony and Jaan for the great initiative!
I am almost done with my first research project on variable selection and classification using a Bayesian forest model – I simplified the variable partition in the model, came up with better tree updates, added a hyperprior, sped up the algorithm by an order of magnitude, and started testing on real data. Among the other ambitious projects of the past year are two MIRIx workshops plus writing up the results, and starting this blog.
Improvements in personal effectiveness:

started using a daily checklist of morning habits
started taking melatonin every night
started tagging new thoughts
started using FollowUpThen to schedule future tasks without overloading my todo list
started using Toggl to track work hours only
stopped using Beeminder (too stressful), and replaced it with a combination of 42goals and FollowUpThen (works well)
quit as President of the Toastmasters club
made a volunteer application form for FLI, so that instead of being inundated with 7 freeform emails per month from interested folks, I get the relevant information in an organized spreadsheet and I’m not required to respond

Random cool things I did:

climbing a 5.11a
climbing outdoors
indoor surfing
polar bear swim

2014 resolutions
A year ago, I made a number of New Year resolutions (and assigned a probability of completion for each goal). Here is how they worked out:
Succeeded:

continue meditation practice (>10 minutes daily, > 120 times) (70%) – did ~200 times
start a new research project (70%) – started working with a new advisor, did some background reading, narrowed down a topic, wrote a grant proposal
do at least 5 pullups in a row (85%)
reading stats blogs
go to 3 conferences, including a MIRI workshop (80%) – went to NESS, JSM, NIPS, and two MIRIx workshops
give 5 speeches at Toastmasters (75%)
give at least 5 LessWrong meetup talks (70%)
run comfort zone expansion outings at least twice (80%)
start a group project at Citadel (40%) – FLI work groups and MIRIx workshops sort of count for this
introduce at least 3 friends to LW meetups (50%)
help people achieve their goals – helped run a weekly habit training session at Citadel

Essentially succeeded:

publish paper about current research project (90%) – almost done, hope to submit by March 1
write at least 5 LW posts (80%) – wrote 4 posts
more writing – reflections (did some), stories (sort of), poems (nope), also journals and blog posts

Failed and no longer endorsed:

continue to avoid Beeminder debt – didn’t work, then stopped using Beeminder, now use 42goals for goal tracking and FollowUpThen for reminders
do consulting for Metamed (60%)
read 90% of the LW Sequences (70%) – made progress, but no longer want to read such a high fraction (waiting for the ebook to come out)
finish Pearl’s Causality (50%) – read the LW review of the book instead
learn more economics and biology

Calibration:

low predictions, 40-60%: 2/4 = 50% (perfectly calibrated)
high predictions, 70-90%: 7/10 = 70% (overconfident)

Conclusions:

Reading goals mostly don’t work for me – if I do set them, flexible goals of the form “read some of X” (like stats blogs) do better than more fixed and time-consuming goals like “read most of X” (like the Sequences)
The rate of goal disendorsement is 5/19 = 26%.
I don’t tend to completely fail on goals I continue to endorse – yay!

2015 goals and habits
Broad categories of goals for the coming year:

Research:

wrap up and submit BFC project,
start and make a significant progress on 1-2 new projects

FLI:

help streamline operations and communication,
continue work on AI safety outreach to AI researchers,
encourage AI safety research,
start projects in risk areas other than AI safety

Self-improvement:

reduce weekend/free-time anxiety,
increase acceptance of suboptimal situations,
improve introspection ability and my model of myself,
improve retention of information (using Anki),
eliminate cluttery speech pattern

Habits to maintain:

daily meditation
daily pushups
daily melatonin
tagging and writing down new thoughts
tracking goals in 42goals
tracking work hours in Toggl
journaling 1-2 times / week

2015 predictions

I will submit the Bayesian Forest Classifier paper for publication (95%)
I will submit another paper besides BFC (40%)
I will present BFC results at a conference (JSM, ICML or NIPS) (40%)
I will get a new external fellowship to replace my expiring NSERC fellowship (50%)
I will skim at least 20 research papers in machine learning (70%)
I will write at least 12 blog posts (70%)
I will climb a 5.12 without rope cheating (50%)
I will lead climb a 5.11a (50%)
I will be able to do 10 pullups in a row (60%)
I will meditate at least 150 times (80%)
I will record at least 150 new thoughts (70%)
I will make at least 100 Anki cards by the end of the year (70%)
I will read at least 10 books (60%)
I will attend Burning Man (90%)
Boston will have a second rationalist house by the end of the year (30%)
FLI will hire a full-time project manager or administrator (80%)
FLI will start a project on biotech safety (70%)

I will update this post if other goals and predictions for the year come to mind before the end of January.

2014-15 New Year review


Open and closed mental states

I learned a game at Burning Man this year that was about connecting to people and reading their nonverbal signals, called the “open-closed” game (h/t Minda Myers). There are two people in the game, and one is trying to approach the other and place a hand on their shoulder. No words can be exchanged, except that person who is being approached can announce their emotional state as “open” or “closed”. When they say “closed”, the approacher may not get any closer until they say “open” again. The approachee monitors themselves for any internal discomfort associated with the other person, and says “closed” if that is the case. The approacher tries to keep the other person comfortable through their body language and eye contact, to get them to remain “open”.
I have recently started playing this game with myself, with “open” representing openness to experience or being in the moment, and “closed” representing tunnel vision or discomfort with the way things are going. In a way, I imagine being “approached” by whatever situation I’m in, or whatever sequence of experiences is happening, instead of a person. I ask myself whether I am in the open or closed state, and try to shift to the open state whenever I notice being in the closed state.

There are a couple of reasons to try to do this. In the open state, I tend to be happier, more curious and observant and have more new thoughts. From a week of tracking my mental states and thought status using TagTime, I can make a preliminary conclusion that while old thoughts do occur in the open state, new thoughts never occur in the closed state. While the closed state makes me more efficient at doing straightforward tasks (e.g. by making me less distractable), it makes me less efficient at doing less straightforward tasks (e.g. by increasing my tendency to optimize locally rather than globally).
This is related to the concept of “againstness” taught by Valentine Smith at the Center for Applied Rationality, which is a sense of resisting something about the situation at hand. Learning to notice this sense more quickly is a valuable thing I learned at CFAR and through my meditation practice. Redirecting attention to body sensations is supposed to be helpful for dissipating againstness, but I have found it difficult to get myself to do this in the moment, and not particularly reliable. Following the driving principle of “focusing on the road and not the curb”, I find it easier to shift to a mental state with a simple salient label like “open” instead of a clunky label like “non-againsty”. It also feels less judgmental to ask myself “what am I closed to right now, what experience am I not letting in?” than “what am I against right now?”.
The againstness approach seems to be about relaxing the mind by relaxing the body first, while for some people relaxing the mind first comes more naturally – I actually find myself automatically breathing deeper when shifting into the open state. For both approaches, the goal is the same – to let go of mental and physical tension before proceeding with what you are doing. The rule of thumb, like in the game, is to first get into the open state and then approach the situation at hand.
(Cross-posted to LessWrong.)

Open and closed mental states


Importance motivation: a double-edged sword

Working on something important is often stressful. A hypothesis came up in conversation (h/t Michael Vassar) that labeling your work as important actually decreases the quality of the output. I have seen this effect in action quite a bit, e.g. when someone can write an essay when commenting on a blogpost, but gets nailed by writer’s block when writing an essay on purpose. Feeling responsible for something important can be an obstacle to productivity when the work requires creativity. How can we do important creative work without being held back by importance considerations?
This has come up in my thesis research project on variable selection. It’s been “almost done” for a year now, and my advisor keeps reminding me that it’s important to finish soon. This motivation results in making little tweaks to the algorithm at the expense of looking at the big picture. However, most of the improvements to the algorithm came about through exploring models that I thought were more interesting than the default one. When I came up with a more streamlined version of the model, for a while the algorithm was doing worse than the original one, and I started cursing myself for following my elegance heuristics instead of doing what needed to be done. Then I found a bug, and the new algorithm reached a similar performance level to the old one. (I do eventually want to stop playing with the model and actually publish the thing, though…)

I would like to distinguish between two types of importance: extrinsic and intrinsic. Extrinsic importance comes from external restrictions, like deadlines, funding availability, user demand or evaluation by high-status people. Intrinsic importance arises from the problem you’re solving – in my research example, this includes criteria like model simplicity and algorithm correctness. This is somewhat of a continuum, since a particular criterion can combine intrinsic and extrinsic importance, for example if my advisor insists on the algorithm being correct. Intrinsic importance fuels interest and curiosity, while extrinsic importance can inhibit creativity.
There is a quote by Howard Thurman: “Don’t ask yourself what the world needs; ask yourself what makes you come alive. And then go and do that. Because what the world needs is people who have come alive.” This can be interpreted as a suggestion to focus on intrinsic rather than extrinsic importance of your work. However, when choosing projects, it’s a good idea to be guided by extrinsic importance at least to some degree, lest you end up working on underwater basket weaving. Ideally, you’d want to find the intersection between the things that the world needs and the things that make you come alive. Throwing out the importance criterion once you start the project seems suboptimal, since it also applies to choosing subproblems within the project.
This can be helped by separating idea generation from idea evaluation, process focus from outcome focus. Write a sloppy draft of the essay, or imagine that you are writing to a close friend, and then step back and edit it based on the needs of your intended audience. Play with a new model for a while, occasionally asking yourself whether to prune this line of inquiry in favor of more promising ones. This process is analogous to a Kalman filter – an algorithm that alternates between an evolution step, where the next step is proposed, and an observation step, where that step is updated using data from the real world. In practice, much more time needs to be spent on developing the next step than on evaluating it, and more importantly, longer chunks of time. You want to correct your course once in a while, instead of stifling good ideas before fully developing them.
There is a saying that I find both annoying and insightful – “life is too important to be taken seriously”. Like many sayings, it’s annoyingly vague, but can be interpreted usefully. My interpretation of choice is that too much evaluation of what you’re doing can make the outcome less good, which conflicts with the purpose of evaluating it in the first place. “Life is too important to be taken seriously too often”, how about that?


Citadel house sessions – a year in review

Since we moved into Citadel House in Boston a year ago, we ran self-improvement and rationality sessions every week, for the housemates and some local LessWrongers. There were around 3 sessions running at a time, and some of them caught on much more than others. I will discuss the sessions in decreasing order of success.
Meta Mondays
Meta Mondays are self-improvement meetings with a theme – most meetings involved a particular activity announced (slightly) in advance, aimed at practicing a skill. Some example activities were:

Feedback-a-thon. A large group of us got on Admonymous, and sent each other feedback messages simultaneously. We had a list of feedback categories on the board (social skills, hygiene, speech patterns, etc), and some people requested feedback in specific categories. We were sufficiently persistent and prolific to overwhelm the website’s email quota!
Table Topics Against Rationality. We combined the idea of Table Topics from Toastmasters (1-2 minute impromptu speeches) with Cards Against Rationality. Everyone took turns drawing two cards from the deck, and then giving a speech that connected the concepts on the two cards to each other. This devolved into silliness at the end, when we threw in the Cards Against Humanity deck.
WRAP decision framework. (“Widen your options, Reality test assumptions, Attain emotional distance, Prepare to be right/wrong.”) I taught this decision procedure, and people played around with applying it, but most of the people present didn’t have a major upcoming decision to practice on.

At times when we didn’t have a theme, we would have a general conversation about what people were optimizing in their lives these days. The themed meetings were generally better attended and more focused, but also required more preparation. In future, we should invite guest hosts for Meta Mondays, instead of coming up with all the topics ourselves.

Order of the Sphex
This is a habit building session, named after the carpenter wasp of CFAR fame, known for stalwartly adhering to its routines. We would go around the circle proposing habits to try for a week, and then on the second round everyone would commit to 1-2 habits to work on. These could be the same habits as last week, sometimes with modifications, or new habits. At the end of the session, I sent everyone involved an email with their commitments.
Due to fluctuating attendance, in practice people followed up on their habits every few weeks instead of weekly. Some found it helpful, others said that talking about a difficult habit without immediately doing something about them made the action more aversive. Depending on why someone is not sticking to habits, this session can be useful if their main failure mode is forgetting or indecision, and counterproductive if the underlying problem is aversion.
Writing
Since many people procrastinate on writing things, we decided to get together and write every week, optionally sharing the writing with others. This session was successful for a while – people wrote blog posts, LessWrong articles, stories, journal entries, emails, etc. Gradually, some regular attendees dropped out, and we started skipping the session on most weeks. Some said that it was hard to write on demand or to finish a piece of writing in a two-hour time block. The usefulness of this session depended on the ability to pick up and continue a previously started piece of writing, which was challenging for many of us.
Goal Factoring
We used the goal factoring technique from CFAR to analyze the motivations behind our actions, and alternative ways to achieve the same goals more cheaply. We also did some aversion factoring – analysis of potentially useful but unpleasant actions. (Here, “action” has a broad meaning that includes things like “thinking about X” or “having emotion X”.) We usually worked by ourselves for a while, and then (optionally) discussed our findings with each other.
One failure mode was people choosing more sharable actions to factor, which were usually less private or embarrassing, and thus likely less useful. We tried to remedy this by holding a “secret goal factoring” session, where people agreed not to share their results. In practice, most people  said they didn’t factor anything particularly private or important in this session, so other barriers to analyzing important actions might be more significant than fear of sharing with a group.
After a few months, we ran out of low hanging fruit in terms of actions to factor, and this session was transformed into Meta Mondays.
Strategic review
For a month or so, I tried to run a weekly CFAR-style strategic review session. This didn’t catch on at all – it was difficult to explain, and since the benefit of the technique depends on regularity, inconsistent attendance made it useless to everyone except me. These days, I do strategic reviews on my own once every few weeks.
All the sessions had the difficulty of staying focused on the activity without devolving into unstructured conversation. This could be helped by running at most two sessions at a time, since there would be more evenings free for random conversation. Another general takeaway is that sessions need a clear purpose that people buy in to.
I am running a survey to see which sessions people found useful last year and/or want to happen this coming year. Stay tuned!