I like this paper, and the techniques for meta-regularization it proposes seem to me like they're natural and will be picked up elsewhere. Penalizing model complexity to encourage more adaptive learning reminds me of arguments that <@pressure for compressed policies could create mesa-optimizers@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) -- this feels like very weak evidence that that could indeed be the case.

Meta-Learning without Memorization

Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, Chelsea Finn

Meta-learning is a technique for leveraging data from previous tasks to enable efficient learning of new tasks. This paper proposes a solution to a problem in meta-learning which the paper calls the _memorization problem_. Imagine a meta-learning algorithm trained to look at 2D pictures of 3D objects and determine their orientation relative to a fixed canonical pose. Trained on a small number of objects, it may be easy for the algorithm to just memorize the canonical pose for each training object and then infer the orientation from the input image. However, the algorithm will perform poorly at test time because it has not seen novel objects and their canonical poses. Rather than memorizing, we would like the meta-learning algorithm to learn to _adapt_ to new tasks, guessing at rules for determining canonical poses given just a few example images of a new object.

At a high level, a meta-learning algorithm uses information from three sources when making a prediction-- the training data, the parameters learned while doing meta-training on previous tasks, and the current input. To prevent memorization, we would like the algorithm to get information about which task it's solving only from the training data, rather than memorizing it by storing it in its other information sources. To discourage this kind of memorization, the paper proposes two new kinds of regularization techniques which it calls "meta-regularization" schemes. One penalizes the amount of information that the algorithm stores in the direct relationship between input data and predicted label ("meta-regularization on activations"), and the other penalizes the amount of information that the algorithm stores in the parameters learned during meta-training ("meta-regularization on weights").

In some cases, meta-regularization on activations fails to prevent the memorization problem where meta-regularization on weights succeeds. The paper hypothesizes that this is because even a small amount of direct information between input data and predicted label is enough to store the correct prediction (e.g., a single number that is the correct orientation). That is, the correct activations will have _low information complexity_, so it is easy to store them even when information in activations is heavily penalized. On the other hand, the _function_ needed to memorize the predicted label has a _high information complexity_, so penalizing information in the weights, which store that function, successfully discourages memorization. The key insight here is that memorizing all the training examples results in a more information-theoretically complex model than task-specific adaptation, because the memorization model is a single model that must simultaneously perform well on all tasks.

Both meta-regularization techniques outperform non-regularized meta-learning techniques in several experimental set-ups, including a toy sinusoid regression problem, the pose prediction problem described above, and modified Omniglot and MiniImagenet classification tasks. They also outperform fine-tuned models and models regularized with standard regularization techniques.


Overall I like this paper's simplicity and focus on understanding how meta learning systems work. I did find the reinforcement learning experiment a bit more difficult to parse and connect to the linear and nonlinear regression experiments, and, of course, there's always the question with work on simpler problems like this of whether the intuition extends to more complex ones

Meta-learners' learning dynamics are unlike learners'

We've seen evidence in prior work that meta learning models can be trained to more quickly learn tasks drawn from some task distribution, by training a model in the inner loop and optimizing against generalization error. This paper suggests that meta learning doesn't just learn new tasks faster, but has a different ordered pattern of how it masters the task. Where a "normal" learner first learns the low-frequency modes (think SGD modes, or Fourier modes) of a simple regularization task, and later the high-frequency ones, the meta learner makes progress on all the modes at the same relative rate. This meta learning behavior seems to theoretically match the way a learner would update on new information if it had the "correct" prior (i.e. the one actually used to generate the simulated tasks).


Meta-Learning: A Survey

This taxonomy of meta-learning classifies approaches by the main type of meta-data they learn from:
1. Evaluations of other models on related tasks
2. Characterisations of the tasks at hand (and a similarity metric between them)
3. The structures and parameters of related models
Vanschoren explores a number of different approaches in each category.


Meta-learning of Sequential Strategies

Pedro A. Ortega, Jane X. Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, Siddhant M. Jayakumar, Tom McGrath, Kevin Miller, Mohammad Azar, Ian Osband, Neil Rabinowitz, András György, Silvia Chiappa, Simon Osindero, Yee Whye Teh, Hado van Hasselt, Nando de Freitas, Matthew Botvinick, Shane Legg

This paper explains theoretically how to structure meta-learning such that it is incentivized to learn optimal solutions to sequence-prediction and decision-making tasks. The core idea is to define a distribution over tasks, and then sample a new task at the beginning of each episode that the agent must then handle. Importantly, the agent is _not told_ what the task is, and so must infer it from observations. As long as you structure the loss function appropriately, the optimal policy for the agent is to maintain a prior over the task that is updated via Bayes Rule after each observation.

Of course, since the agent is actually a neural net with memory, it does not explicitly perform Bayes Rule, but rather learns a set of weights that instantiate an update rule that effectively approximates Bayes Rule for the given task distribution. Since this update rule only needs to work on the specific task distribution being meta-trained on, it can be made significantly more efficient than a full-blown Bayes Rule, and thus can be learned by a relatively small neural net. We can think of this as the network implementing a full-blown _reasoning process_.

In the case of sequence prediction, we optimize the log probability assigned to the true outcomes. As a simple example, the agent might observe a sequence of coin flips from a single coin, where the bias of that coin is chosen at the beginning of each episode (and is not given to the agent). If the bias is drawn from a Normal distribution centered at 0.5, the agent will start out predicting 50-50 on Heads/Tails; if it then sees a Heads, it might update slightly to something like 55-45, and vice versa for Tails. In contrast, if the bias is drawn from a distribution where most of the mass is near 0 or 1, and very little mass is at 0.5, the agent will still start out predicting 50-50, but after seeing a Heads it will then update strongly to e.g. 90-10.

In the case of sequential decision-making, we are given a reward function; we simply optimize the expected reward using some traditional deep RL algorithm (the paper considers Q-learning).


I suspect that in these simple tasks the posterior distribution over the parameters θ maintained by the Bayes-optimal algorithm is a _minimal_ sufficient statistic, that is, _any_ optimal policy must have states that are sufficient to reconstruct the information stored by the Bayes-optimal algorithm. So it makes sense that, for an agent with optimal behavior, the agent’s states could be used to simulate the Bayes-optimal states. I don’t think this tells us that much about the algorithm the network is implementing.

Note that I am quite happy to see work investigating the sorts of reasoning processes that neural networks have learned. While I don’t think the specific results in this paper have told us that much, I’m excited to see this line of work scaled up to more complex tasks, where agents may not reach optimal behavior, or might do so by learning heuristics that _don’t_ encode all of the information that the Bayes-optimal algorithm would use.

Paper: Meta-trained agents implement Bayes-optimal agents

Understanding meta-trained algorithms through a Bayesian lens

Vladimir Mikulik*, Grégoire Delétang*, Tom McGrath*, Tim Genewein*, Miljan Martic, Shane Legg, Pedro A. Ortega

The previous paper suggested that meta-learning can implement optimal reasoning processes in theory. Does it work in practice? This paper sets out to answer this question by studying some simple prediction and decision-making tasks.

For prediction, we consider agents that are trained on a family of distributions (e.g. Bernoulli distributions whose parameter is chosen from a Beta distribution) to predict the probability distribution after seeing a sample generated from it. For decision-making, we consider two-armed bandit problems (where again there is a distribution over the parameters of the problem). These problems were chosen because their optimal solutions can be calculated analytically.

The authors train neural nets with memory to perform well on these tasks (as discussed in the previous paper) and find that they do indeed behave optimally, achieving effectively the best possible performance. They then try to investigate whether they are implementing the same reasoning algorithm as the analytic Bayes-optimal solution. To do this, they see whether they can train a second neural net to map the hidden states (memory) of the agent to the states in the Bayes-optimal solution, and vice versa. (One way to think of this: can you simulate the Bayes-optimal algorithm using the observation encodings from the RNN, and vice versa?)

They find that they _can_ learn a good mapping from agent states to Bayes-optimal states, but _cannot_ learn a good mapping from Bayes-optimal states to agent states. It seems likely that the agent has states that encode more information than is necessary, and so the minimal information stored by the Bayes-optimal algorithm is insufficient to reconstruct the agent states.


I like how this paper analyzes the loss landscape of a particular problem, multi-task learning, and uses that knowledge to derive a new algorithm. One thing I always find tricky in ML papers is that it is hard to establish that the theory of why an algorithm works (typically shown on toy models) is also the reason it improves performance (typically shown using complex neural networks). I appreciate that this paper checks for the conditions of their theorem in the multi-task RL models that they train. That said, I think that in order to confirm that the tragic triad they describe is the mechanism by which *PCGrad* improves performance, they would require some way to toggle each element of the triad while keeping everything else fixed.

Gradient Surgery for Multi-Task Learning

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn

In multi-task learning, an algorithm is given data from multiple tasks and tries to learn them all simultaneously, ideally sharing information across them. This paper identifies a *tragic triad* of conditions that can prevent gradient descent from finding a good minimum when all three are present:

**Conflicting gradients** occur when the gradient from one task points in a different direction from another.
**Dominating gradients** occur when the gradient from one task is much larger in magnitude than another.
**High curvature** is when the multi-task curvature is high in the direction of the gradient.

In this situation, the linear approximation of the gradient to the high curvature area leads to an overestimation of the increase in performance on the dominant gradient's task and an underestimation of the performance degradation from the conflicting gradient's task. I find picturing the parabola y=x^2 and seeing that a gradient descent step overestimates progress while a gradient ascent step underestimates to be helpful in understanding this.

To solve this, they propose *PCGrad*, which projects all gradients into the normal plane of the others in a pairwise fashion. Their theoretical analysis establishes convergence properties of *PCGrad*, and they empirically show it can be combined with other multi-task algorithms to improve performance and that it makes optimization easier for multi-task supervised learning and RL. They also show plots confirming that the necessary conditions for their theorems appear in these contexts.


Especially given the <@growth of compute@>(@AI and Compute@), this agenda seems like a natural one to pursue to get AGI. Unfortunately, it also mirrors very closely the phenomenon of <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), with the only difference being that it is _intended_ that the method produces a powerful inner optimizer. As the paper acknowledges, this introduces several risks, and so it calls for deep engagement with AI safety researchers (but sadly it does not propose ideas on how to mitigate the risks).

Due to the vast data requirements, most of the environments would have to be simulated. I suspect that this will make the agenda harder than it may seem at first glance -- I think that the complexity of the real world was quite crucial, and that simulating environments that reach the appropriate level of complexity will be a very difficult task. (My intuition is that something like <@Neural MMO@> is nowhere near enough complexity.)

AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

Historically, the <@bitter lesson@>(@The Bitter Lesson@) has been that approaches that leverage increasing computation for learning outperform ones that build in a lot of knowledge. The current ethos towards AGI seems to be that we will come up with a bunch of building blocks (e.g. convolutions, transformers, trust regions, GANs, active learning, curricula) that we will somehow manually combine into one complex powerful AI system. Rather than require this manual approach, we could instead apply learning once more, giving the paradigm of AI-generating algorithms, or AI-GA.

AI-GA has three pillars. The first is to **learn architectures**: this is analogous to a superpowered neural architecture search that can discover convolutions, recurrence and attention without any hardcoding. The second is to **learn the learning algorithms**, i.e. meta-learning. The third and most underexplored pillar is to learn to **generate complex and diverse environments** within which to train our agents. This is a natural extension of meta-learning: with meta-learning, you have to specify the distribution of tasks the agent should perform well on; AI-GA simply says to learn this distribution as well. <@POET@>(@Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions@) is an example of recent work in this area.

A strong reason for optimism about the AI-GA paradigm is that it mimics the way that humans arose: natural selection was a very simple algorithm that with a _lot_ of compute and a very complex and diverse environment was able to produce a general intelligence: us. Since it would need fewer building blocks (since it aims to learn everything), it could succeed faster than the manual approach, at least if the required amount of compute is not too high. It is also much more neglected than the "manual" approach.

However, there are safety concerns. Any powerful AI that comes from an AI-GA will be harder to understand, since it's produced by this vast computation where everything is learned, and so it would be hard to get an AI that is aligned with our values. In addition, with such a process it seems more likely that a powerful AI system "catches us by surprise" -- at some point the stars align and the giant computation makes one good random choice and suddenly it outputs a very powerful and sample efficient learning algorithm (aka an AGI, at least by some definitions). There is also the ethical concern that since we'd end up mimicking evolution, we might accidentally instantiate large amounts of simulated beings that can suffer (especially if the environment is competitive, as was the case with evolution).


I believe that I found this paper the _very first time_ I looked for generic new interesting papers _after_ I started thinking about this problem, which was quite the coincidence. I'm really happy that the authors wrote the paper -- it's not in their incentives (as far as I can tell), but the topic seems crucial to address.

That said, I disagree with the paper on a few counts. The authors don't acknowledge the value of HARKing -- **often it is useful to run many experiments and see what happens in order to develop a good theory**. Humans are not ideal Bayesian reasoners who can consider all hypotheses at once; we often require many observations in order to even hypothesize a theory. The authors make the point that in other fields HARKing leads to bad results, but ML is significantly different in that **we can run experiments much faster with a much higher iteration speed**.

If we were instead forced to preregister studies, as the authors suggest, **the iteration speed would drop by an order of magnitude or two**; I seriously doubt that the benefits would outweigh the cost of lower iteration speed. Instead of preregistering all experiments, maybe researchers could run experiments and observe results, formulate a theory, and then preregister an experiment that would test the theory -- but in this case I would expect that researchers end up "preregistering" experiments that are very similar to the experiments that generated the theory, such that the results are very likely to come out in support of the theory.

(This does not require any active malice on the part of the researchers -- it's natural to think of predictions of the theory in the domain where you developed the theory. For example, in <@our recent paper@>(@Learning Preferences by Looking at the World@), we explicitly designed four environments where we expected our method to work and one where it wouldn't.)

Another point: I think that **the underlying cause of HARKing is the incentive to chaise SotA**, and if I were writing this paper I would focus on that. For example, I believe that the bias towards SotA chasing causes HARKing, and not the other way around. (I'm not sure if the authors believe otherwise; the paper isn't very clear on this point.) This is also a more direct explanation of results being caused by grad student descent or hyperparameter tuning; the HARKing in such papers occur because it isn't acceptable to say "we obtained this result via grad student descent", because that would not be a contribution to the field.

Although I've been critiquing the paper, overall I find my beliefs much closer to the authors' than the "beliefs of the field". (Not the beliefs of researchers in the field: I suspect many researchers would agree that HARKing has negative effects, even though the incentives force researchers to do so in order to get papers published.) I'd be interested in exploring the topic further, but don't have enough time to do so myself -- if you're interested in building toy models of the research field and modeling the effect of interventions on the field, get in touch and we can see if it would make sense to collaborate.

HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning

Oguzhan Gencoglu, Mark van Gils, Esin Guldogan, Chamin Morikawa, Mehmet Süzen, Mathias Gruber, Jussi Leinonen, Heikki Huttunen

This paper focuses on the negative effects of Hypothesizing After the Results are Known (HARKing), a pattern in which researchers **first conduct experiments and view the results**, and once they have hit the bar to be publishable, **a hypothesis is constructed after the fact to explain the results**. It argues that HARKing is common in machine learning, and that this has negative effects on the field as a whole. First, improvements to state-of-the-art (SotA) may be questionable because they could have been caused by sufficient hyperparameter tuning via grad student descent, instead of the new idea in a paper to which the gain is attributed. Second, there is publication bias since only positive results are reported in conferences, which prevents us from learning from negative results. Third, hypotheses that are tailored to fit results for a single dataset or task are much less likely to generalize to new datasets or tasks. Fourth, while AutoML systems achieve good results, we cannot figure out what makes them work because the high compute requirements make ablation studies much harder to perform. Finally, they argue that we need to fix HARKing in order to achieve things like ethical AI, human-centric AI, reproducible AI, etc.


These summaries seem really good to me (probably higher quality than a typical summary that I write), but are often on topics I'm not an expert in (eg. GANs) so it's hard for me to evaluate. The one paper I knew well ([Inverse Reward Design](https://arxiv.org/abs/1711.02827)) had a good summary.

ML Writing Month May 2018

The author wrote up a summary of an ML paper every day in May, which have all been collected in this doc.


I really like this paper from the point of view of illustrating an alternative paradigm to building powerful AI systems that _isn’t_ based on scaling up neural networks. You might have picked up from the last few newsletters that I generally do expect us to build powerful AI systems by scaling up neural networks, so you might expect that I disagree with this paper. This is only partially true. I do in fact think that many of the skills mentioned in this paper will emerge by training very large neural networks on diverse datasets; indeed we’re already seeing this with <@few-shot learning@>(@Language Models are Few-Shot Learners@). However, this likely only happens at what would be truly mind-boggling amounts of compute today: in order for this to be remotely feasible, we need to have <@exponential improvements in hardware cost and algorithmic efficiency@>(@Draft report on AI timelines@). It is plausible to me that some of the needed improvements in algorithmic efficiency will come through ideas similar to the ones in this paper: for example, just as CNNs provided a useful inductive bias of translation-invariance, perhaps we get a new architecture that has an inductive bias towards compositionality or causality.

Building Machines That Learn and Think Like People

Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman

The core claim of this 2016 paper is that we should focus on building AI systems that work as _flexibly_ as humans do. For example, a human can learn how to play the Atari game Frostbite in just a couple of hours, way faster than typical deep RL algorithms -- and in addition, after this they will likely be able to transfer zero-shot to new reward functions, such as "lose as quickly as possible", "maximize the number of fish", "beat the level with as little time to spare as possible", and so on. How can we build AI systems that mimic this feat? Deep RL certainly doesn't get us there. Similarly, while neural networks can learn to classify digits and characters with thousands of examples, humans can learn new characters from a single example, which then allows them to perform many different tasks such as classification, generation, parsing it into different pen strokes, etc. Since the paper was written neural nets have made progress on few-shot classification, but are still quite far from the flexibility that humans display.

You might reasonably object that humans have rich priors built from years of lived experience, as well as innate knowledge baked in by evolution; in contrast, a neural network has to learn from scratch. The authors agree: in their view, the challenge is **how to imbue rich priors into artificial agents**, so that they too can exhibit these impressive behaviors that humans show. Their preferred approach is to take inspiration from human learning and intelligence as much as possible. In this paper, they identify three main ingredients to recreate that flexibility, and provide an overview of the existing literature:

1. **Developmental software:** This refers to the basic capabilities that children have, even before they learn language. These are called "intuitive theories" in cognitive science; think of "intuitive physics" and "intuitive psychology" theories.

2. **Model building:** Neural networks primarily work via _pattern matching_, but in order to get human-level flexibility, you will need to build _models_: this enables flexibility because the same model can be used for a variety of different tasks. (For example, you can reuse your understanding of the environment transitions in Frostbite when the reward function changes.) Models need to be _compositional_, that is, the representations should be capable of being composed with each other to provide new semantically meaningful representation. For example, for handwritten characters, the representation of a character should be the composition of the representations of the individual pen strokes used to make the character. The authors also highlight _causality_ and _learning to learn_ as important.

3. **Thinking fast:** One major drawback of models is that getting _conclusions_ from these models often requires slow, complex inference algorithms. But human thinking is actually quite fast; just think of how quickly we can understand a visual scene. How can we get this property as well? First, we can use approximate inference algorithms to get answers much more quickly (in fact, one line of work distills the inference algorithm into a fast neural network for even more speed). Second, we can combine model-based and model-free algorithms together; for example we might use a model-based algorithm for flexibility but then use the data generated by that algorithm to train a model-free method that can run faster.


I really like that this paper not just claims that there is a problem with reproducibility, but demonstrates this more rigorously using an experiment. More robust empirical findings seem quite important for getting to a better understanding of machine learning systems in the medium term. Since this understanding is especially important for safety relevant research, where exploratory research seems more problematic by default, I am excited for a push in that direction.

Unreproducible Research is Reproducible

Xavier Bouthillier, Cesar Laurent, Pascal Vincent

This paper argues that despite the growing popularity of sharing code, machine learning research has a problem with reproducibility. It makes the distinction between the reproducibility of **methods/results**, which can be achieved by fixing random seeds and sharing code, and the reproducibility of **findings/conclusions**, which requires that different experimental setups (or at least random seeds) lead to the same conclusion.

Several popular neural network architectures are trained on several image classification datasets several times with different random seeds determining the weight initialization and sampling of data. The relative rankings of the architectures with respect to the test accuracy are found to vary relevantly with the random seed for all data sets, as well as between data sets.

The authors then argue that while the reproducibility of methods can help with speeding up **exploratory research**, the reproducibility of findings is necessary for **empirical research** from which robust conclusions can be drawn.  They claim that exploratory research that is not based on robust findings can get inefficient, and so call for the machine learning community to do more empirical research.


The Bitter Lesson

This blog post is controversial. This is a combination summary and opinion, and so is more biased than my summaries usually are.

Much research in AI has been about embedding human knowledge in AI systems, in order to use the current limited amount of compute to achieve some outcomes. That is, we try to get our AI systems to think the way we think we think. However, this usually results in systems that work currently, but then cannot leverage the increasing computation that will be available. The bitter lesson is that methods like search and learning that can scale to more computation eventually win out, as more computation becomes available. There are many examples that will likely be familiar to readers of this newsletter, such as chess (large scale tree search), Go (large scale self play), image classification (CNNs), and speech recognition (Hidden Markov Models in the 70s, and now deep learning).

Shimon Whiteson's [take](https://twitter.com/shimon8282/status/1106534178676506624) is that in reality lots of human knowledge has been important in getting AI to do things; such as the invariances built into convolutional nets, or the MCTS and self-play algorithm underlying AlphaZero. I don't see this as opposed to Rich Sutton's point -- it seems to me that the takeaway is that we should aim to build algorithms that will be able to leverage large amounts of compute, but we can be clever and embed important knowledge in such algorithms. I think this criterion would have predicted ex-ante (i.e. before seeing the results) that much past and current research in AI was misguided, without also predicting that any of the major advances (like CNNs) were misguided.

It's worth noting that this is coming from a perspective of aiming for the most general possible capabilities for AI systems. If your goal is to instead build something that works reliably now, then it really is a good idea to embed human domain knowledge, as it does lead to a performance improvement -- you should just expect that in time the system will be replaced with a better performing system with less embedded human knowledge.

One disagreement I have is that this post doesn't acknowledge the importance of data. The AI advances we see now are ones where the data has been around for a long time (or you use simulation to get the data), and someone finally put in enough engineering effort + compute to get the data out and put it in a big enough model. That is, currently compute is <@increasing much faster@>(@AI and Compute@) than data, so the breakthroughs you see are in domains where the bottleneck was compute and not data; that doesn't mean data bottlenecks don't exist.


This is well presented and you can glean a lot from looking at the introductory sections. If you choose to dive into a particular topic, charts and methodology are presented in a clear manner with nice hyperlinking to make navigation relatively painless. There is also an [interactive](https://aiindex.stanford.edu/vibrancy/) visualization that allows for cross-country comparison according to user-defined metrics. Once again, very well presented.

2021 AI Index Report

Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham, Jack Clark, Raymond Perrault

The AI Index Report is a project to track and distill data related to artificial intelligence. One central theme the report focuses on is the effects of COVID on AI research direction. The report highlights significant increases in spending on drug development, 4.5 times that in 2019. The report also focuses a spotlight on the relative lack of AI ethics benchmarks. This could pose a significant problem as surveillance technologies become an increasingly mature technology. Beyond these broad themes, there's data on publication trends, politics, diversity, and more in the 222-page report. Additionally, a significant amount of data is publicly available or interactive.


I enjoyed and appreciated the way this paper questions the obvious default of convolutional models for image processing, and in particular the way it highlights various reachable points (neighborhood-aware value transformation, relative position encoding, etc) on the interpolation path between weights based purely on relative distance (convolution) and weights based purely on content similarity (attention without any position representation). I'd be interested in the future to see more work in this space, exploring different places in a network architecture where content-focused or position-focused computation is most valuable.

Stand-Alone Self Attention in Vision Models

Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jonathon Shlens

Continuing with the more general rise of attention models across disciplines, this paper argues that attention-only models can perform comparably to convolutional networks on image classification tasks, a domain where convolution has been the reigning default method for years now.  Because attention doesn't scale parameter-wise as you increase spatial scale, this comparable performance can be achieved at a notably lower number of parameters and FLOPs. The authors perform a few interesting modifications to attention. Firstly, it's canonical, with attention, to include a representation of a pixel's position in the image, in addition to the vector storing the content of the image. In this paper, they found that storing this position information in relative terms (i.e. "how close is this pixel to the center one where attention is being calculated") performs better.

This can be seen as a sort of generalized form of convolution, where instead of having fixed weights for pixels in a kernel indexed by their relative position, attention takes both content and relative position as an input and generates a weight dynamically. Another modification is, at the lower parts of the network, to somewhat modify the attention paradigm such that the "value" at each location isn't just a neutrally transformed version of the input at that location, but rather one transformed differently according to the pixel's position relative to the anchor point where attention is being calculated. At the lower levels of the network, convolutions tend to outperform attention, but attention performs better at later layers of the network. This makes sense, the authors claim, because in early layers each individual pixel doesn't contain much content information that an attention mechanism could usefully leverage, whereas later the learned features at a given spatial location are richer, and more productively leveraged by attention.


Sparsity seems like a good objective to shoot for in order to ensure explainability. I'm less convinced that it's worthwhile for representation learning: I doubt humans have any sort of "sparse learning" bias; I think sparsity of knowledge is a natural consequence of having to understand a very complex world with a very small brain. (Whereas current ML systems only have to understand much simpler environments.)

Explainable AI, Sparse Representations, and Signals

So far, we have built AI systems that store knowledge _symbolically_ or in a _distributed fashion_ (with neural nets being the latter). While the distributed form allows us to learn knowledge and rules automatically, it is much harder to understand and interpret than symbolically represented knowledge. This post argues that the main difference is in the **sparsity** of the learned knowledge. Of course, with more "sparse" knowledge, it should be easier for us to understand the internal workings of the AI system, since we can ignore the pruned connections. However, the author also argues that sparse knowledge will help 'guide the search for models and agents that can be said to "learn" but also "reason"'. Given that AGI will likely involve finding good representations for the world (in the sense of unsupervised learning), then sparse learning can be thought of as a bias towards finding better [bases](https://en.wikipedia.org/wiki/Basis_(linear_algebra)) for world models, that are more likely to be conceptually clean and more in line with Occam's razor.

In a postscript, the author considers arguments for AI risk. Notably, there isn't any consideration of goal-directedness or alignment failures; the worry is that we will start applying superhuman AI systems to superhuman tasks, and we won't know how to deal with these situations.


I imagine that this is primarily targeted at data scientists aiming to learn about the real world, and not ML researchers, but I wouldn't be surprised if it was helpful for us as well. MNIST and ImageNet are both present, and a search for "self-driving cars" turned up some promising-looking links that I didn't investigate further.

Making it easier to discover datasets

Google has launched Dataset Search, a tool that lets you search for datasets that you could then use in research.


In <@last year’s report@>(@State of AI Report 2020@), I said that their 8 predictions seemed to be going out on a limb, and that even 67% accuracy woud be pretty impressive. This year, they scored their predictions as 5 “Yes”, 1 “Sort of”, and 2 “No”. That being said, they graded “The first 10 trillion parameter dense model” as “Yes”, I believe on the basis that Microsoft had run a couple of steps of training on a 32 trillion parameter dense model. I definitely interpreted the prediction as saying that a 10 trillion parameter model would be trained _to completion_, which I do not think happened publicly, so I’m inclined to give it a “No”. Still, this does seem like a decent track record for what seemed to me to be non-trivial predictions. This year's predictions seem similarly "out on a limb" as last year's.

This year’s report included one-slide summaries of many papers I’ve summarized before. I only found one major issue -- the slide on <@TruthfulQA@>(@TruthfulQA: Measuring How Models Mimic Human Falsehoods@) implies that larger language models are less honest _in general_, rather than being more likely to imitate human falsehoods. This is actually a pretty good track record, given the number of things they summarized where I would have noticed if there were major issues.

State of AI Report 2021

As with <@past@>(@State of AI@) <@reports@>(@State of AI Report 2020@), I'm not going to summarize the entire thing; instead you get the high-level themes that the authors identified:

1. AI is stepping up in more concrete ways, including in mission critical infrastructure.
2. AI-first approaches have taken biology by storm (and we aren't just talking about AlphaFold).
3. Transformers have emerged as a general purpose architecture for machine learning in many domains, not just NLP.
4. Investors have taken notice, with record funding this year into AI startups, and two first ever IPOs for AI-first drug discovery companies, as well as blockbuster IPOs for data infrastructure and cybersecurity companies that help enterprises retool for the AI-first era.
5. The under-resourced AI-alignment efforts from key organisations who are advancing the overall field of AI, as well as concerns about datasets used to train AI models and bias in model evaluation benchmarks, raise important questions about how best to chart the progress of AI systems with rapidly advancing capabilities.
6. AI is now an actual arms race rather than a figurative one, with reports of recent use of autonomous weapons by various militaries.
7. Within the US-China rivalry, China's ascension in research quality and talent training is notable, with Chinese institutions now beating the most prominent Western ones.
8. There is an emergence and nationalisation of large language models.


The article does a reasonably good job of giving equal attention to different viewpoints on neural-symbolic integration. While the article does focus on the localist vs. distributed distinction, I also find it to be broadly useful. Personally, after reading the article I wonder if 'reasoning' needs to be hand-set into a neural network at all. Is there really something inherently different about reasoning such that it wouldn't just emerge from any sufficiently powerful forward predictive model? The authors make a good point regarding XAI and the importance of fidelity. I agree that it's important that our explanations specifically fit the model rather than interpret the data. However, from a performance perspective, I don't feel I have a good understanding of why the abstraction of a symbol/logic should occur outside the neural network. This leaves me thinking [**the bitter lesson**](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) ([**AN #49**](https://mailchi.mp/efed27be268a/alignment-newsletter-49)) will apply to neural-symbolic approaches that try to extract symbols or apply reason using human features (containers/first-order logic).

**Rohin's opinion:** While I do think that you can get human-level reasoning (including e.g. causality) by scaling up neural networks with more diverse data and environments, this does _not_ mean that neural-symbolic methods are irrelevant. I don’t focus on them much in this newsletter because 1) they don’t seem that relevant to AI alignment in particular (just as I don’t focus much on e.g. neural architecture search) and 2) I don’t know as much about them, but this should not be taken as a prediction that they won’t matter. I agree with Zach that the bitter lesson will apply, in the sense that for a specific task as we scale up we will tend to reproduce neural-symbolic approaches with end-to-end approaches. However, it could still be the case that for the most challenging and/or diverse tasks, neural-symbolic approaches will provide useful knowledge / inductive bias that make them the best at a given time, even though vanilla neural nets could scale better (if they had the data, memory and compute).

[Logic Tensor Networks](https://arxiv.org/abs/1606.04422), [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)

Neurosymbolic AI: The 3rd Wave

The field of neural-symbolic AI is broadly concerned with how to combine the power of discrete symbolic reasoning with the expressivity of neural networks. This article frames the relevance of neural-symbolic reasoning in the context of a big question: what are the necessary and sufficient building blocks of AI? The authors address this and argue that AI needs to have both the ability to learn from and make use of experience. In this context, the neural-symbolic approach to AI seeks to establish provable correspondences between neural models and logical representations. This would allow neural systems to generalize beyond their training distributions through neural-reasoning and would constitute significant progress towards AI.

The article surveys the last 20 years of research on neural-symbolic integration. As a survey, a number of different perspectives on neural-symbolic AI are presented. In particular, the authors tend to see neural-symbolic reasoning as divided into two camps: localist and distributed. Localist approaches assign definite identifiers to concepts while distributed representations make use of continuous-valued vectors to work with concepts. In the later parts of the article, promising approaches, current challenges, and directions for future work are discussed.

Recognizing 'patterns' in neural networks constitutes a localist approach. This relates to explainable AI (XAI) because recognizing how a given neural model makes a decision is a pre-requisite for interpretability. One justification for this approach is that codifying patterns in this way allows systems to avoid reinventing the wheel by approximating functions that are already well-known. On the other hand, converting logical relations (if-then) into representations compatible with neural models constitutes a distributed approach. One distributed method the authors highlight is the conversion of statements in first-order logic to vector embeddings. Specifically, Logic Tensor Networks generalize this method by grounding logical concepts onto tensors and then using these embeddings as constraints on the resulting logical embedding.

Despite the promising approaches to neural-symbolic reasoning, there remain many challenges. Somewhat fundamentally, formal reasoning systems tend to struggle with existential quantifiers while learning systems tend to struggle with universal quantification. Thus, the way forward is likely a combination of localist and distributed approaches. Another challenging area lies in XAI. Early methods for XAI were evaluated according to fidelity: measures of the accuracy of extracted knowledge in relation to the network rather than the data. However, many recent methods have opted to focus on explaining data rather than the internal workings of the model. This has resulted in a movement away from fidelity which the authors argue is the wrong approach.


State of AI Report 2020

The third <@State of AI@> report is out! I won't go into details here since there is really quite a lot of information, but I recommend scrolling through the presentation to get a sense of what's been going on. I was particularly interested in their 8 predictions for the next year: most of them seemed like they were going out on a limb, predicting something that isn't just "the default continues". On last year's 6 predictions, 4 were correct, 1 was wrong, and 1 was technically wrong but quite close to being correct;  even this 67% accuracy would be pretty impressive on this year's 8 predictions. (It does seem to me that last year's predictions were more run-of-the-mill, but that might just be hindsight bias.)


I really liked the super detailed description of a large number of things that humans can do but current vision systems cannot do; it feels like I have a much more detailed sense now of what is missing from current approaches to vision. While the paper has a huge 491 references backing up its claims, I’m not sure how relevant all of them are. For example, the reference to the revelation principle didn’t really seem to justify the associated point. As a counterpoint, the discussion on utility functions in various fields was excellent. Unfortunately I’m not familiar enough with most of the other areas to spot check them.

I read this paper because I heard that the last author, Song-Chun Zhu, was leaving his job as a professor at UCLA to set up a research institute on “general AI” in Beijing, and I wanted to get a sense of what the institute was likely to work on. It seems like the institute will probably pursue an agenda that focuses on building the five particular facets of intelligence into AI systems, as a form of inductive bias: this is how you’d get to a “small data for big tasks” paradigm. If that’s right, it would be in stark contrast to the neural network approaches taken by most of industry, and the biology-inspired approaches taken by (say) the Human Brain Project, but it would feel quite aligned with the views of many academics (like Josh Tenenbaum, who is a coauthor on this paper).

Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Joshua B. Tenenbaum, Song-Chun Zhu

This paper argues that current computer vision research focuses too much on a "big data for small tasks" paradigm that focuses only on the "what" and "where" of images. More work should be done on a "small data for big tasks" paradigm that focuses more on the "how" and "why" of images. These "how" and "why" questions focus attention on details of an image that may not be directly present in the pixels of the image, which the authors term "dark" data (analogously to dark matter in physics, whose existence is inferred, not observed). For example, by asking why a human is holding a kettle with the spout pointing down, we can infer that the kettle contains liquid that will soon come out of the kettle, even though there are no pixels that directly correspond to the liquid.

The authors propose five important areas for further research, abbreviated FPICU, and do a literature review within each one:
1. **Functionality:** Many objects, especially those designed by humans, can be better understood by focusing on what functionalities they have.
2. **Physics:** Cognitive science has shown that humans make extensive use of _intuitive physics_ to understand the world. For example, simply reasoning about whether objects would fall can provide a lot of constraints on a visual scene; it would be weird to see an upright cup floating in the air.
3. **Intent:** The world is filled with goal-directed agents, and so understanding the world requires us to infer the goals that various agents have. This is a capability humans get very quickly -- at eighteen months of age, children can infer and imitate the intended goal of an action, even if the action fails to achieve the goal.
4. **Causality:** Much has already been written about causality; I will not bore you with it again. The authors see this as the most important factor that underlies the other four areas.
5. **Utility:** I didn't really get how this differed from intent. The section in the paper discusses utility theory, and then talks about work that infers utility functions from behavior.


I've summarized the post pretty well, I think -- mainly I'd encourage you to play the Think Again game, it's pretty fun and you generate useful data for them.

Predicting Slow Judgments

Andreas Stuhlmueller, Owain Evans, Tom McGrath, Zac Kenton, Chris Cundy, Ryan Carey, Andrew Schreiber, Neal Jean, Girish Sastry

A joint project between Ought and FHI. The goal is to predict judgments that humans make after considering a question for many hours. However, that data is expensive to collect, so they also collect a lot of labels of how humans make judgments in limited time, and use those as noisy labels for the slow, deliberative judgments. They have just released Think Again (https://thinkagain.ought.org/), where you can make fast and slow judgments on Fermi estimates, political statements, or ML papers. They are especially looking for people to make judgments on ML papers. This will take an hour or two, but they'll recommend papers that they think you'll like based on your judgments of the papers they showed you.


I enjoyed this book more than I thought I would: it had a lot of novel content for me, and I liked the explanations and comparisons across different kinds of existential risks (something that I hadn't really seen a single unified perspective on), and I especially liked the constant focus on what we do and don't know -- it felt more like a research paper (albeit in a conversational style) than a popular book, and was similarly information-dense.

On the AI part specifically, I liked that one of the endnotes cashed out powerful AI systems using model-based RL: this indeed seems like the thing that is closest to the classic expected utility maximizer, so the conclusions make a bit more sense. You still have to wonder how exactly the model is learned, and how exactly the AI system becomes good at using the model to find good actions, but at least under those two assumptions you would have all the standard convergent instrumental subgoals. In contrast, with model-free RL, the default expectation is that the RL agent needs to try things multiple times before it can learn to do them again, so it's less clear how it starts doing novel things. It seems that model-based and model-free RL are pretty similar so the distinction doesn't matter in practice, but at least conceptually it's a lot easier to reason about the model-based system (at least in the context of AI risk).

Toby gives a 1 in 10 chance of existential catastrophe from AI in the next century (more than half of his total of 1 in 6), which decomposes into a 1 in 2 chance of AGI this century, and 1 in 5 of it leading to existential catastrophe. This is a bit more pessimistic than Paul's <@estimate@>(@Conversation with Paul Christiano@) of 10% EV loss (which was over all time, not just this century), which is in turn a bit more pessimistic than the 1 in 10 chance that I <@estimated@>(@Conversation with Rohin Shah@) (and am now forever anchored on), which was over all time _and_ conditional on no additional effort from longtermists. But I wouldn't read too much into this -- 10 is a nice round number, and that probably played a big role in why I chose it. I certainly don't feel calibrated enough to easily tell the difference between 1 in 5 and 1 in 20 on a question of this complexity.

I am very happy about this trend of people actually stating numbers: it's a lot easier to narrow down on the important disagreements when people put down numbers, even if they're completely made up. I'd really like to see numbers from people who have larger disagreements (as I expect would be the case with e.g. MIRI researchers).

FLI Podcast: The Precipice: Existential Risk and the Future of Humanity with Toby Ord

This book argues that humanity is in a special stage of its development: it is on the _precipice_, a narrow time during which we have enough power to destroy ourselves, but not enough wisdom to have mitigated such risks. It first argues that existential risk would be very important to reduce (for all the standard reasons), and then considers many different kinds of existential risks, finding that natural ones (asteroids, supervolcanoes, stellar explosions) are small relative to anthropogenic risks, both current (nuclear war, climate change, environmental destruction) and future (engineered pandemics, unaligned AI, dystopian scenarios). I'll focus primarily on the part about AI risk, as well as some of the comments on existential risk in general.

The AI risk presentation in the book was similar to that in [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/<PHONE>): it argues for risk from goal-directed AI systems (though the terminology used in the book is different). It first demonstrates the strong progress in deep learning, and then notes that expert surveys estimate that AGI is more likely than not to arrive in the next century. It then notes that we don't know how to specify a reward function for an AI system (even with e.g. inverse reinforcement learning), and to the extent that we get it wrong, it pits us in competition against a superintelligent adversary. Ideas like switching off the AI system wouldn't work, due to convergent instrumental subgoals like survival.

It also considers some obvious objections, including the very reasonable objection that "AI researchers won't build something that will kill them". However, Toby is still worried, citing that due to the unilateralist curse unaligned AGI might still be built by the most optimistic researchers, and in any case the personal benefits to the researchers might justify the risk of misalignment to them personally (though it would not be justified for the world as a whole).

The book then spends some time discussing _risk factors_, which are things that do not _directly_ lead to existential risks, but indirectly _exacerbate_ other existential risks, making them more likely. For example, great power war seems like a risk factor: it isn't going to cause an existential catastrophe by itself, but it increases the likelihood that we use risky technologies like bioweapons and AI that could then cause an existential catastrophe.

The book also has lots of useful insights about existential risks in general, which then also apply to AI risk: for example, risks that strike sooner should be prioritized (since the later risks can be dealt with later), risks that are more sudden will be more important to focus on (since we won't be able to build support as the risk gradually comes in), and risks that are "sharper" will be more neglected since there won't be as many "warning shots".


I'm excited to see more big-picture thought about AI alignment out of DeepMind. This newsletter (and I) tend to focus a lot more on the technical alignment problem than the normative one, partly because there's more work on it, but also partly because I think it is the [more urgent problem](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) (a [controversial](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#JK9Jzvz8f4BEjmNqi) [position](https://www.alignmentforum.org/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty)).

Artificial Intelligence, Values and Alignment

This paper from a DeepMind author considers what it would mean to align an AI system. It first makes a distinction between the _technical_ and _normative_ aspects of the AI alignment problem. Roughly, the normative aspect asks, "what should our AI systems do?", while the technical aspect asks, "given we know what our AI systems should do, how do we get them to do it?". The author argues that these two questions are interrelated and should not be solved separately: for example, the current success of deep reinforcement learning in which we _maximize expected reward_ suggests that it would be much easier to align AI to a utilitarian framework in which we _maximize expected utility_, as opposed to a deontological or Kantian framework.

The paper then explores the normative aspect, in both the single human and multiple humans case. When there's only one human, we must grapple with the problem of what to align our AI system to. The paper considers six possibilities: instructions, expressed intentions, revealed preferences, informed preferences, interests, and values, but doesn't come to a conclusion about which is best. When there are multiple humans, we must also deal with the fact that different people disagree on values. The paper analyzes three possibilities: aligning to a global notion of morality (e.g. "basic human rights"), doing what people would prefer from behind a veil of ignorance, and pursuing values that are determined by a democratic process (the domain of social choice theory).

See also [Import AI #183](https://jack-clark.net/2020/02/03/import-ai-183-curve-fitting-conversation-with-meena-gans-show-us-our-climate-change-future-and-what-compute-data-arbitrage-means/)


I look forward to this post every year. This year, it's been a stark demonstration of how much work _doesn't_ get covered in this newsletter -- while I tend to focus on the technical alignment problem, with some focus on AI governance and AI capabilities, Larks's literature review spans many organizations working on existential risk, and as such has many papers that were never covered in this newsletter. Anyone who wants to donate to an organization working on AI alignment and/or x-risk should read this post. However, if your goal is instead to figure out what the field has been up to for the last year, for the sake of building inside view models of what's happening in AI alignment, I _might_ soon write up such an overview myself, but no promises.

2019 AI Alignment Literature Review and Charity Comparison

As in [three](https://forum.effectivealtruism.org/posts/nSot23sAjoZRgaEwa/2016-ai-risk-literature-review-and-charity-comparison) [previous](https://forum.effectivealtruism.org/posts/XKwiEpWRdfWo7jy7f/2017-ai-safety-literature-review-and-charity-comparison) <@years@>(@2018 AI Alignment Literature Review and Charity Comparison@), this mammoth post goes through the work done within AI alignment from December 2018 - November 2019, from the perspective of someone trying to decide which of several AI alignment organizations to donate to. As part of this endeavor, Larks summarizes several papers that were published at various organizations, and compares them to their budget and room for more funding.


I strongly agree with 2 and 4, weakly agree with 1, 5, and 6, and disagree with 3.

I'm Buck Shlegeris, I do research and outreach at MIRI, AMA

Here are some beliefs that Buck reported that I think are particularly interesting (selected for relevance to AI safety):

1. He would probably not work on AI safety if he thought there was less than 30% chance of AGI within 50 years.
2. The ideas in <@Risks from Learned Optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) are extremely important.
3. If we build "business-as-usual ML", there will be inner alignment failures, which can't easily be fixed. In addition, the ML systems' goals may accidentally change as they self-improve, obviating any guarantees we had. The only way to solve this is to have a clearer picture of what we're doing when building these systems. _(This was a response to a question about the motivation for MIRI's research agenda, and so may not reflect his actual beliefs, but just his beliefs about MIRI's beliefs.)_
4. Different people who work on AI alignment have radically different pictures of what the development of AI will look like, what the alignment problem is, and what solutions might look like.
5. Skilled and experienced AI safety researchers seem to have a much more holistic and much more concrete mindset: they consider a solution to be composed of many parts that solve subproblems that can be put together with different relative strengths, as opposed to searching for a single overall story for everything.
6. External criticism seems relatively unimportant in AI safety, where there isn't an established research community that has already figured out what kinds of arguments are most important.


I enjoyed reading this book; I don't usually get to read a single person's overall high-level view on the state of AI, how it could have societal impact, the argument for AI risk, potential solutions, and the need for AI governance. It's nice to see all of these areas I think about tied together into a single coherent view. While I agree with much of the book, especially the conceptual switch from the standard model of intelligent machines to Stuart's model of beneficial machines, I'm going to focus on disagreements in this opinion.

First, the book has an implied stance towards the future of AI research that I don't agree with: I could imagine that powerful AI systems end up being created by learning alone without needing the conceptual breakthroughs that Stuart outlines. This has been proposed in e.g. <@AI-GAs@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@)), and seems to be the implicit belief that drives OpenAI and DeepMind's research agendas. This leads to differences in risk analysis and solutions: for example, the <@inner alignment problem@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) only applies to agents arising from learning algorithms, and I suspect would not apply to Stuart's view of AI progress.

The book also gives the impression that to solve AI safety, we simply need to make sure that AI systems are optimizing the right objective, at least in the case where there is a single human and a single robot. Again, depending on how future AI systems work, that could be true, but I expect there will be other problems that need to be solved as well. I've already mentioned inner alignment; other graduate students at CHAI work on e.g. [robustness](https://adversarialpolicies.github.io/) and transparency.

The proposal for aligning AI requires us to build a model that relates human preferences to human behavior. This sounds extremely hard to get completely right. Of course, we may not need a model that is completely right: since reward uncertainty makes the agent amenable to shutdowns, it seems plausible that we can correct mistakes in the model as they come up. But it's not obvious to me that this is sufficient.

The sections on multiple humans are much more speculative and I have more disagreements there, but I expect that is simply because we haven't done enough research yet. For example, HC worries that we won't be able to use laws to prevent AIs from doing technically legal but still antisocial things for the benefit of a single human. This seems true if you imagine that a single human suddenly gets access to a superintelligent AI, but when everyone has a superintelligent AI, then the current system where humans socially penalize each other for norm violations may scale up naturally. The overall effect depends on whether AI makes it easier to violate norms, or to detect and punish norm violations.

Human Compatible: Artificial Intelligence and the Problem of Control

_Since I am aiming this summary for people who are already familiar with AI safety, my summary is substantially reorganized from the book, and skips large portions of the book that I expect will be less useful for this audience. If you are not familiar with AI safety, **note that I am skipping many arguments and counterarguments in the book that are aimed for you**. I'll refer to the book as "HC" in this newsletter._

Before we get into details of impacts and solutions to the problem of AI safety, it's important to have a model of how AI development will happen. Many estimates have been made by figuring out the amount of compute needed to run a human brain, and figuring out how long it will be until we get there. HC doesn't agree with these; it suggests the bottleneck for AI is in the algorithms rather than the hardware. We will need several conceptual breakthroughs, for example in language or common sense understanding, cumulative learning (the analog of cultural accumulation for humans), discovering hierarchy, and managing mental activity (that is, the metacognition needed to prioritize what to think about next). It's not clear how long these will take, and whether there will need to be more breakthroughs after these occur, but these seem like necessary ones.

What could happen if we do get beneficial superintelligent AI? While there is a lot of sci-fi speculation that we could do here, as a weak lower bound, it should at least be able to automate away almost all existing human labor. Assuming that superintelligent AI is very cheap, most services and many goods would become extremely cheap. Even many primary products such as food and natural resources would become cheaper, as human labor is still a significant fraction of their production cost. If we assume that this could bring up everyone's standard of life up to that of the 88th percentile American, that would result in nearly a _tenfold_ increase in world GDP per year. Assuming a 5% discount rate per year, this corresponds to $13.5 _quadrillion_ net present value. Such a giant prize removes many reasons for conflict, and should encourage everyone to cooperate to ensure we all get to keep this prize.

Of course, this doesn't mean that there aren't any problems, even with AI that does what its owner wants. Depending on who has access to powerful AI systems, we could see a rise in automated surveillance, lethal autonomous weapons, automated blackmail, fake news and behavior manipulation. Another issue that could come up is that once AI is better than humans at all tasks, we may end up delegating everything to AI, and lose autonomy, leading to _human enfeeblement_.

This all assumes that we are able to control AI. However, we should be cautious about such an endeavor -- if nothing else, we should be careful about creating entities that are more intelligent than us. After all, the gorillas probably aren't too happy about the fact that their habitat, happiness, and existence depends on our moods and whims. For this reason, HC calls this the _gorilla problem_: specifically, "the problem of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence". Of course, we aren't in the same position as the gorillas: we get to _design_ the more intelligent "species". But we should probably have some good arguments explaining why our design isn't going to succumb to the gorilla problem. This is especially important in the case of a fast intelligence explosion, or _hard takeoff_, because in that scenario we do not get any time to react and solve any problems that arise.

Do we have such an argument right now? Not really, and in fact there's an argument that we _will_ succumb to the gorilla problem. The vast majority of research in AI and related fields assumes that there is some definite, known _specification_ or _objective_ that must be optimized. In RL, we optimize the _reward function_; in search, we look for states matching a _goal criterion_; in statistics, we minimize _expected loss_; in control theory, we minimize the _cost function_ (typically deviation from some desired behavior); in economics, we design mechanisms and policies to maximize the _utility_ of individuals, _welfare_ of groups, or _profit_ of corporations. This leads HC to propose the following standard model of machine intelligence: _Machines are intelligent to the extent that their actions can be expected to achieve their objectives._ However, if we put in the wrong objective, the machine's obstinate pursuit of that objective would lead to outcomes we won't like.

Consider for example the content selection algorithms used by social media, typically maximizing some measure of engagement, like click-through. Despite their lack of intelligence, such algorithms end up changing the user's preference so that they become more predictable, since more predictable users can be given items they are more likely to click on. In practice, this means that users are pushed to become more extreme in their political views. Arguably, these algorithms have already caused much damage to the world.

So the problem is that we don't know how to put our objectives inside of the AI system so that when it optimizes its objective, the results are good for us. Stuart calls this the "King Midas" problem: as the legend goes, King Midas wished that everything he touched would turn to gold, not realizing that "everything" included his daughter and his food, a classic case of a <@badly specified objective@>(@Specification gaming examples in AI@). In some sense, we've known about this problem for a long time, both from King Midas's tale, and in stories about genies, where the characters inevitably want to undo their wishes.

You might think that we could simply turn off the power to the AI, but that won't work, because for almost any definite goal, the AI has an incentive to stay operational, just because that is necessary for it to achieve its goal. This is captured in what may be Stuart's most famous quote: _you can't fetch the coffee if you're dead_. This is one of a few worrisome [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf).

What went wrong? The problem was the way we evaluated machine intelligence, which doesn't take into account the fact that machines should be useful for _us_. HC proposes: _Machines are **beneficial** to the extent that **their** actions can be expected to achieve **our** objectives_. But with this definition, instead of our AI systems optimizing a definite, wrong objective, they will _also_ be uncertain about the objective, since we ourselves don't know what our objectives are. HC expands on this by proposing three principles for the design of AI systems, that I'll quote here in full:

1. _The machine's only objective is to maximize the realization of human preferences._
2. _The machine is initially uncertain about what those preferences are._
3. _The ultimate source of information about human preferences is human behavior._

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) provides a formal model of an _assistance game_ that showcases these principles. You might worry that an AI system that is uncertain about its objective will not be as useful as one that knows the objective, but actually this uncertainty is a feature, not a bug: it leads to AI systems that are deferential, that ask for clarifying information, and that try to learn human preferences. [The Off-Switch Game](https://arxiv.org/abs/1611.08219) shows that because the AI is uncertain about the reward, it will let itself be shut off. These papers are discussed later in this newsletter.

So that's the proposed solution. You might worry that the proposed solution is quite challenging: after all, it requires a shift in the entire way we do AI. What if the standard model of AI can deliver more results, even if just because more people work on it? Here, HC is optimistic: the big issue with the standard model is that it is not very good at learning our preferences, and there's a huge economic pressure to learn preferences. For example, I would pay a lot of money for an AI assistant that accurately learns my preferences for meeting times, and schedules them completely autonomously.

Another research challenge is how to actually put principle 3 into practice: it requires us to connect human behavior to human preferences. [Inverse Reward Design](https://arxiv.org/abs/1711.02827) and <@Preferences Implicit in the State of the World@>(@Learning Preferences by Looking at the World@) are example papers that tackle portions of this. However, there are _lots_ of subtleties in this connection. We need to use _Gricean semantics_ for language: when we say X, we do not mean the literal meaning of X: the agent must also take into account the fact that we bothered to say X, and that we didn't say Y. For example, I'm only going to ask for the agent to buy a cup of coffee if I believe that there is a place to buy reasonably priced coffee nearby. If those beliefs happen to be wrong, the agent should ask for clarification, rather than trudge hundreds of miles or pay hundreds of dollars to ensure I get my cup of coffee.

Another problem with inferring preferences from behavior is that humans are nearly always in some deeply nested plan, and many actions don't even occur to us. Right now I'm writing this summary, and not considering whether I should become a fireman. I'm not writing this summary because I just ran a calculation showing that this would best achieve my preferences, I'm doing it because it's a subpart of the overall plan of writing this bonus newsletter, which itself is a subpart of other plans. The connection to my preferences is very far up. How do we deal with that fact?

There are perhaps more fundamental challenges with the notion of "preferences" itself. For example, our _experiencing self_ and our _remembering self_ may have different preferences -- if so, which one should our agent optimize for? In addition, our preferences often change over time: should our agent optimize for our current preferences, even if it knows that they will predictably change in the future? This one could potentially be solved by learning _meta-preferences_ that dictate what kinds of preference change processes are acceptable.

All of these issues suggest that we need work across many fields (such as AI, cognitive science, psychology, and neuroscience) to reverse-engineer human cognition, so that we can put principle 3 into action and create a model that shows how human behavior arises from human preferences.

So far, we've been talking about the case with a single human. But of course, there are going to be multiple humans: how do we deal with that? As a baseline, we could imagine that every human gets their own agent that optimizes for their preferences. However, this will differentially benefit people who care less about other people's welfare, since their agents have access to many potential plans that wouldn't be available to an agent for someone who cared about other people. For example, if Harriet was going to be late for a meeting with Ivan, her AI agent might arrange for Ivan to be even later.

What if we had laws that prevented AI systems from acting in such antisocial ways? It seems likely that superintelligent AI would be able to find loopholes in such laws, so that they do things that are strictly legal but still antisocial, e.g. line-cutting. (This problem is similar to the problem that we can't just write down what we want and have AI optimize it.)

What if we made our AI systems utilitarian (assuming we figured out some acceptable method of comparing utilities across people)? Then we get the "Somalia problem": agents will end up going to Somalia to help the worse-off people there, and so no one would ever buy such an agent.

Overall, it's not obvious how we deal with the transition from a single human to multiple humans. While HC focuses on a potential solution for the single human / single agent case, there is still much more to be said and done to account for the impact of AI on all of humanity. To quote HC, "There is really no analog in our present world to the relationship we will have with beneficial intelligent machines in the future. It remains to be seen how the endgame turns out."


AI Alignment Podcast: Human Compatible: Artificial Intelligence and the Problem of Control

This podcast covers some of the main ideas from the book, which I'll ignore for this summary. It also talks a bit about the motivations for the book. Stuart has three audiences in mind. He wants to explain to laypeople what AI is and why it matters. He wants to convince AI researchers that they should be working in this new model of beneficial AI that optimizes for our objectives, rather than the standard model of intelligent AI that optimizes for its objectives. Finally, he wants to recruit academics in other fields to help connect human behavior to human preferences (principle 3), as well as to figure out how to deal with multiple humans.

Stuart also points out that his book has two main differences from Superintelligence and Life 3.0: first, his book explains how existing AI techniques work (and in particular it explains the standard model), and second, it proposes a technical solution to the problem (the three principles).


Alignment Newsletter One Year Retrospective

The Alignment Newsletter is one year old! I've written a retrospective of the newsletter's impact over the last year, with a lot of open questions about what the newsletter should look like in the future. Please help me figure out the answers by *taking this 3-minute [survey](https://docs.google.com/forms/d/e/1FAIpQLSepm8ATtun9pvLGL9MXhk6Z59hO14huEVS6qgqtl3KS6n5SIQ/viewform?usp=sf_link)*, and if you're feeling particularly generous with your time, read the retrospective and tell me your opinions in the comments!


The exponential blowup in potential loss with multiple actions would make this prohibitive, but of course you could instead view the full sequence of actions (i.e. trajectory) as a mega-action, and quantilize over this mega-action. In this case, a one-millionth-quantilizer could choose from among the million best plans that a human would make (assuming a well-specified U), and any unintended consequences (that were intentionally chosen by the quantilizer) would have to be ones that a human had a one-in-a-million chance of causing to occur, which quite plausibly excludes really bad outcomes.

Phrased this way, quantilization feels like an amplification of a human policy. Unlike the amplification in iterated amplification, it does _not_ try to preserve alignment, it simply tries to bound how far away from alignment the resulting policy can diverge. As a result, you can't iterate quantilization to get arbitrarily good capabilities. You might hope that humans could learn from powerful AI systems, grow more capable themselves (while remaining as safe as they were before), and then the next quantilizers would be more powerful.

It's worth noting that the theorem in the paper shows that, to the extent that you think quantilization is insufficient for AI alignment, you need to make some other assumption, or find some other source of information, in order to do better, since quantilization is optimal for its particular setup. For example, you could try to assume that U is at least somewhat reasonable and not pathologically bad; or you could assume an interactive setting where the human can notice and correct for any issues with the U-maximizing plan before it is executed; or you could not have U at all and exceed human performance through some other technique.

I'm not very worried about the issue that quantilization could forgo benefits that the human policy had. It seems that even if this happens, we could notice this, turn off the quantilizer, and fix the utility function U so that it no longer ignores those benefits. (We wouldn't be able to prevent the quantilizer from forgoing benefits of our policy that we didn't know about, but that seems okay to me.)

[Quantilizers: A Safer Alternative to Maximizers for Limited Optimization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) and [When to use quantilization](https://www.alignmentforum.org/posts/Rs6vZCrnQFWQ4p37P/when-to-use-quantilization)

A key worry with AI alignment is that if we maximize expected utility for some utility function chosen by hand, we will likely get unintended side effects that score highly by the utility function but are nevertheless not what we intended. We might hope to leverage human feedback to solve this: in particular, an AI system that simply mimics human actions would often be desirable. However, mimicry can only achieve human performance, and cannot improve upon it. The first link is a 2015 paper that introduces quantilization, which interpolates between these two extremes to improve upon human performance while bounding the potential (expected) loss from unintended side effects.

In particular, let's suppose that humans have some policy γ (i.e. probability distribution over actions). We evaluate utility or performance using a utility function U, but we _do not assume_ it is well-specified -- U can be any function, including one we would not want to maximize. Our goal is to design a policy π that gets higher expected U than γ (reflecting our hope that U measures utility well) without doing too much worse than γ in the worst case when U was as badly designed as possible. We'll consider a one-shot case: π is used to select an action once, and then the game is over.

The core idea behind quantilization is simple: if our policy only does things that the human might have done, any expected loss it incurs corresponds to some loss that the human could incur. So, let's take our human policy γ, keep only the top q-fraction of γ (as evaluated by U), and then sample an action from there. This defines our policy π_q, also called a q-quantilizer. For example, suppose the human would choose A with probability 0.25, B with probability 0.5, and C with probability 0.25, and U(A) > U(B) > U(C). Then a (1/4)-quantilizer would choose A with certainty, a (1/2)-quantilizer would choose randomly between A and B, and a (3/8)-quantilizer would choose A twice as often as B.

Note that even if π_q places all of its probability on the worst possible actions (because U was badly designed), by construction the human had probability q of doing the same thing as π_q, and so the expected cost of π_q can be at most (1/q) times larger than the expected cost of the human policy γ. In fact, if we have no other knowledge, a q-quantilizer maximizes expected utility U subject to the constraint of never doing worse than (1/q) times as bad as γ.

As soon as you move to the setting with multiple actions, if you choose each action using quantilization, then your worst case bound is exponential in the number of actions. If you assume the cost for each action is independent, you recover the guarantees, but this is not a realistic assumption (as the authors note). Long-term plans are very good or very bad _because_ all of the actions build on each other to achieve some goal, so the costs are not going to be independent.

The second link is a recent post that further analyzes quantilization. It points out that the theoretical analysis in the paper assumes that deviating from the human policy can only lead to costs. If the deviation could also forgo benefits, then the theorems no longer apply. Perhaps the bottom (1-q) of actions actually have strong positive benefits, that we failed to model in U. Then by eliminating those actions, we may have lost arbitrary amounts of expected value. In these situations, the only way to bound the expected regret is by exact mimicry. It also points out that if you are aiming to simultaneously do well both on U and the worst-case bound, then typically imitation will be better since adding any optimization can drastically weaken the worst-case bound and usually will not make U better by the same amount. Quantilization makes sense when there is a "sweet-spot of actions that are pretty common but substantially outperform imitation".


This post, and its accompanying paper, seems very sensible to me. While I have some doubts about how informative human debate data will be about superhuman debaters, it certainly seems worth trying to gain more empirical information. Note that while the paper primarily discusses Debate, I think that many of its arguments are applicable to any human-in-the-loop safety methods (and probably others too). Currently I think Ought is the safety group focusing most on collecting human data, but I look forward to seeing other researchers doing so.

AI Safety Needs Social Scientists

One approach to AI safety is to "ask humans a large number of questions about what they want, train an ML model of their values, and optimize the AI system to do well according to the learned values". However, humans give answers that are limited, biased and often in disagreement with each other, and so AI safety needs social scientists to figure out how to improve this data - which eventually may be gathered from thousands or millions of people. Of particular importance is the ability to design rigorous experiments, drawing from an interdisciplinary understanding of human cognition and behaviour. The authors discuss [Debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)) as a case study of a safety technique whose success depends on empirical questions such as: how skilled are humans as judges by default? Can we train people to be better judges? Are there ways to restrict debate to make it easier to judge?

There are a couple of key premises underlying this argument. The first is that, despite human biases, there are correct answers to questions about human values - perhaps defined as the answer we would endorse if given all relevant information and unlimited time to think. However, it's not necessary for AIs to always find those answers, as long as they are able to recognise cases in which they're uncertain and do nothing (while there are some cases in which inaction can cause harm, such as a self-driving car ceasing to steer mid-journey, it seems that the most worrying long-term catastrophes can be avoided by inaction). Another reason for optimism is that even incomplete or negative results from social science experiments may be useful in informing technical safety research going forward. However, in some cases the systems we're trying to reason about are very different from anything we can test now - for example, AI debaters that are much stronger than humans.


I don't think that the two methods are very different, and I expect that if you can control for "optimization power", the two methods would be about equally susceptible to Goodhart's Law. (In any given experiment, one will be better than the other, for reasons that depend on the experiment, but averaged across experiments I don't expect to see a clear winner.) However, I do think that gradient descent is very powerful at optimization, and it's hard to imagine the astronomically large random search that would compare with it, and so in any practical application gradient descent will lead to more Goodharting (and more overfitting) than random search. (It will also perform better, since it won't underfit, as random search would.)

One of the answers to this question talks about some experimental evidence, where they find that they can get different results with a relatively minor change to the experimental procedure, which I think is weak evidence for this position.

How does Gradient Descent Interact with Goodhart?

Scott often thinks about optimization using a simple proxy of "sample N points and choose the one with the highest value", where larger N corresponds to more powerful optimization. However, this seems to be a poor model for what gradient descent actually does, and it seems valuable to understand the difference (or to find out that there isn't any significant difference). A particularly interesting subquestion is whether [Goodhart's Law](https://www.alignmentforum.org/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) behaves differently for gradient descent vs. random search.


I don't currently think very much about timelines, intelligence explosions, and other questions that AI Impacts thinks about, but it seems very plausible to me that these could be extremely important. (I do think about discontinuities in progress and am very glad I read the [AI Impacts post](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/) on the subject.) One point that the interview brings up is that there are very few (perhaps two?) full time equivalents working on predicting the future of AI, while there are many people working on technical AI safety, so the former is more neglected. I'm not sure I agree with this -- the number of full time equivalents doing technical AI alignment research seems quite small (on the order of 50 people). However, I do see _many_ people who are trying to skill up so that they can do technical AI alignment research, and none who want to do better prediction, and that seems clearly wrong. I would guess that there are several readers of this newsletter who want to do technical AI alignment research, but who would have more impact if they worked in an adjacent area, such as prediction as at AI Impacts, or policy and strategy work, or in better tools and communication. Even though I'm well-placed to do technical research, I still think that common knowledge of research is a big enough bottleneck that I spend a lot of time on this newsletter. It seems likely that there is someone else who would do a better job than me, but who is set on technical safety research even though they wouldn't be as good. So I guess if you are still trying to figure out how to best help with AI alignment, or are about to start training up to do technical research, please do listen to this podcast and consider that alternative route, and various others as well. The goal is not to figure out which question is the most important, so that you can try to solve it. You'll likely do better by considering the field as a whole, and asking which area you would be in if someone optimally assigned people in the field to tasks.

80K podcast with Katja Grace

Rob Wiblin interviewed Katja Grace of AI Impacts about her work predicting the future of AI. My main takeaway was that there are many important questions in this space that almost no one is trying to answer, and that we haven't made a good enough attempt yet to conclude that it's too hard to do, so we should put more time into it. If you haven't seen AI Impacts' work before, you can get some of the most interesting results (at a high level) from listening to this podcast. There's a ton of detail in the podcast -- too much for me to summarize here.


I especially liked this series for the argument that 2% economic growth very likely cannot last much longer, providing quite a strong argument for the importance of this century, without relying at all on controversial facts about AI. At least personally I was previously uneasy about how “grand” or “extraordinary” AGI claims tend to be, and whether I should be far more skeptical of them as a result. I feel significantly more comfortable with these claims after seeing this argument.

Note though that it does not defuse all such uneasiness -- you can still look at how early we appear to be (given the billions of years of civilization that could remain in the future), and conclude that the simulation hypothesis is true, or that there is a Great Filter in our future that will drive us extinct with near-certainty. In such situations there would be no extraordinary impact to be had today by working on AI risk.

The "most important century" series

In some sense, it is really weird for us to claim that there is a non-trivial chance that in the near future, we might build [transformative AI](https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence) and either (1) go extinct or (2) exceed a growth rate of (say) 100% per year. It feels like an extraordinary claim, and thus should require extraordinary evidence. One way of cashing this out: if the claim were true, this century would be the most important century, with the most opportunity for individuals to have an impact. Given the sheer number of centuries there are, this is an extraordinary claim; it should really have extraordinary evidence. This series argues that while the claim does seem extraordinary, _all_ views seem extraordinary -- there isn't some default baseline view that is "ordinary" to which we should be assigning most of our probability.

Specifically, consider three possibilities for the long-run future:
1. **Radical:** We will have a productivity explosion by 2100, which will enable us to become technologically mature. Think of a civilization that sends spacecraft throughout the galaxy, builds permanent settlements on other planets, harvests large fractions of the energy output from stars, etc.
2. **Conservative:** We get to a technologically mature civilization, but it takes hundreds or thousands of years. Let's say even 100,000 years to be ultra conservative.
3. **Skeptical:** We never become technologically mature for some reason. Perhaps we run into fundamental technological limits, or we choose not to expand into the galaxy, or we're in a simulation, etc.
It's pretty clear why the radical view is extraordinary. What about the other two?

The conservative view implies that we are currently in the most important 100,000-year period. Given that life is billions of years old, and would presumably continue for billions of years to come once we reach a stable galaxy-wide civilization, that would make this the most important 100,000 year period out of tens of thousands of such periods. Thus the conservative view is also extraordinary, for the same reason that the radical view is extraordinary (albeit it is perhaps only half as extraordinary as the radical view).

The skeptical view by itself does not seem obviously extraordinary. However, while you could assign 70% probability to the skeptical view, it seems unreasonable to assign 99% probability to such a view -- that suggests some very strong or confident claims about what prevents us from colonizing the galaxy, which we probably shouldn't have given our current knowledge. So, we need to have a non-trivial chunk of probability on the other views, which still opens us up to critique of having extraordinary claims.

Okay, so we've established that we should at least be willing to say something as extreme as "there's a non-trivial chance we're in the most important 100,000-year period". Can we tighten the argument, to talk about the most important _century_? In fact, we can, by looking at the economic growth rate.

You are probably aware that the US economy grows around 2-3% per year (after adjusting for inflation), so a business-as-usual, non-crazy, default view might be to expect this to continue. You are probably also aware that exponential growth can grow _very_ quickly. At the lower end of 2% per year, the economy would double every ~35 years. If this continued for 8200 years, **we'd need to be sustaining multiple economies as big as today's entire world economy _per atom in the galaxy_**. While this is not a priori impossible, it seems quite unlikely to happen. This suggests that we're in one of fewer than 82 centuries that will have growth rates at 2% or larger, making it far less "extraordinary" to claim that we're in the most important one, especially if you believe that growth rates are well correlated with change and ability to have impact.

The actual radical view that the author places non-trivial probability on is one we've seen before in this newsletter: it is one in which there is  automation of science and technology through advanced AI or whole brain emulations or other possibilities. This allows technology to substitute for human labor in the economy, which produces a positive feedback loop as the output of the economy is ploughed back into the economy creating superexponential growth and a "productivity explosion", where the growth rate increases far _beyond_ 2%. The series summarizes and connects together <@many@>(@Modeling the Human Trajectory@), <@past@>(@Could Advanced AI Drive Explosive Economic Growth?@), <@Open@>(@Draft report on AI timelines@), <@Phil@>(@How Much Computational Power It Takes to Match the Human Brain@) <@analyses@>(@Semi-informative priors over AI timelines@), which I won't be summarizing here (since we've summarized these analyses previously). While this is a more specific and "extraordinary" claim than even the claim that we live in the most important century, it seems like it should not be seen as so extraordinary given the arguments above.

This series also argues for a few other points important to longtermism, which I'll copy here:
1. **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.
2. **The long-run future could come much faster than we think**, due to a possible AI-driven productivity explosion. (I briefly mentioned this above, but the full series devotes much more space and many more arguments to this point.)
3. We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**