Alethea Power
Yuri Burda
Harri Edwards
Igor Babuschkin
Vedant Misra

GROKKING: GENERALIZATION BEYOND OVERFIT-TING ON SMALL ALGORITHMIC DATASETS

INTRODUCTION The generalization of overparameterized neural networks has long been a source of interest to the machine learning community since it defies intuitions derived from classical learning theory. In this paper we show that training networks on small algorithmically generated datasets can reliably exhibit unusual generalization patterns, clearly decoupled from performance on the training set, in a significantly more pronounced way than such effects manifest on datasets derived from natural data (see Figure  1 , left, for an example). Such experiments can be quickly reproduced on a single GPU, and this makes them convenient testbeds for theories of generalization. The datasets we consider are binary operation tables of the form a • b = c where a, b, c are discrete symbols with no internal structure, and • is a binary operation. Examples of binary operations include Figure  1 : Left. Grokking: A dramatic example of generalization far after overfitting on an algorithmic dataset. We train on the binary operation of division mod 97 with 50% of the data in the training set. Each of the 97 residues is presented to the network as a separate symbol, similar to the representation in the figure to the right. The red curves show training accuracy and the green ones show validation accuracy. Training accuracy becomes close to perfect at < 10 3 optimization steps, but it takes close to 10 6 steps for validation accuracy to reach that level, and we see very little evidence of any generalization until 10 5 steps. Center. Training time required to reach 99% validation accuracy increases rapidly as the training data fraction decreases. Right. An example of a small binary operation table. We invite the reader to make their guesses as to which elements are missing. addition, composition of permutations, and bivariate polynomials. Training a neural network on a proper subset of all possible equations then amounts to filling in the blanks of the binary op table, much like solving a Sudoku puzzle. An example is shown on the right in Figure  1 . Since we use distinct abstract symbols for all distinct elements a, b, c involved in the equations, the network is not made aware of any internal structure of the elements, and has to learn about their properties only from their interactions with other elements. For example the network doesn't see numbers in decimal notation, or permutations in line notation. Our contributions are as follows: • We show that neural networks are capable of generalizing to the empty slots in a variety of binary op tables. • We show that, long after severely overfitting, validation accuracy sometimes suddenly begins to increase from chance level toward perfect generalization. We call this phenomenon 'grokking'. An example is shown in Figure  1 . • We present the data efficiency curves for a variety of binary operations. • We show empirically that the amount of optimization required for generalization quickly increases as the dataset size decreases. • We compare various optimization details to measure their impact on data efficiency. We find that weight decay is particularly effective at improving generalization on the tasks we study. • We visualize the symbol embeddings learned by these networks and find that they sometimes uncover recognizable structure of the mathematical objects represented by the symbols.
 METHOD All of our experiments used a small transformer trained on datasets of equations of the form a • b = c, where each of "a", "•", "b", "=", and "c" is a separate token. Details of the operations studied, the architecture, training hyperparameters and tokenization can be found in Appendix A.1.
 EXPERIMENTS
 GENERALIZATION BEYOND OVERFITTING Deep learning practitioners are used to seeing small improvements in validation accuracy after validation loss stops decreasing. A double descent of validation loss has been documented in some circumstances, but is considered unusual among practitioners  (Nakkiran et al., 2019; Belkin et al., 2018; d'Ascoli et al., 2020) . On the small algorithmic datasets that we study, improved generalization after initial overfitting occurs for a range of models, optimizers, and dataset sizes, and in some cases these effects are extremely pronounced. A typical example is shown for modular division in Figure  1 . There we see that validation accuracy starts increasing beyond chance level only after 1000 times more optimization steps than are required for training accuracy to get close to optimal. In Figure  4  the training/validation losses are also plotted and we see the double descent of the validation loss. We found these behaviors to be typical for all the binary operations for dataset sizes that were close to the minimal dataset size for which the network generalized within the allotted optimization budget. For larger dataset sizes, the training and validation curves tend to track each other more closely.
 LEARNING TIME CURVES In a typical supervised learning problem, decreasing the amount of training data decreases the converged generalization performance of the model when the optimization procedure is capable of interpolating the training data. In our setting, we observe a different phenomenon: while the converged performance stays constant at 100% within a range of training dataset sizes, the optimization time required to achieve that performance grows quicky as the dataset size is decreased. Figure  1  (center) shows median number of optimization steps until validation performance first reaches 99% for the product in abstract group S 5 . In the vicinity of 25-30% of data, a decrease of 1% of training data leads to an increase of 40-50% in median time to generalization. While the number of steps until validation accuracy > 99% grows quickly as dataset size decreases, the number of steps until the train accuracy first reaches 99% generally trends down as dataset size decreases and stays in the range of 10 3 -10 4 optimization steps. We've observed a similar pattern of exponential increase in optimization time until reaching generalization as dataset size decreases on all the algorithmic tasks for which we could get the networks to generalize.
 GROKKING ON A VARIETY OF PROBLEMS We've measured the mean accuracy across three runs for training datasets consisting of different fractions of all available equations for a variety of binary operations listed in Appendix A.1.1. The results are presented in Figure  2  (right). Since the operands are presented to the neural network as unrelated abstract symbols, the operations x + y (mod p − 1) and x * y (mod p) with a prime number p and non-zero x, y are indistinguishable from the neural network's perspective (and similarly x − y (mod p − 1) and x/y (mod p)). This is because every nonzero residue modulo a prime can be represented as a power of a primitive root. This representation shows the equivalence (up to renaming of symbols) of modular addition modulo p − 1 and modular multiplication modulo p. We see in Figure  2  (right) that x − y and x/y indeed take about the same amount of data for generalization to occur. Some of the operations listed in Figure  2  (right) are symmetric with respect to the order of the operands (x + y, x * y, x 2 + y 2 and x 2 + xy + y 2 ). Such operations tend to require less data for generalization than closely related non-symmetrical counterparts (x − y, x/y, x 2 + xy + y 2 + x). We believe this effect might be partially architecture-dependent, since it's easy for a transformer to learn a symmetric function of the operands by ignoring positional embedding. Some operations (for example x 3 + xy 2 + y (mod 97)) didn't lead to generalization within the allowed optimization budget at any percentage of data up to 95%. The converged models effectively just memorized the training dataset without finding any real patterns in the data. To such a model, the data is effectively random. The operation [x/y (mod p) if y is odd, otherwise x − y (mod p)] requires the network to learn a mix of several simple operations -in particular the role of x has to be interpreted as a residue in the additive group when it's paired with an even y, and as a residue in the multiplicative group when it's paired with an odd y. This shows that generalization can happen even for operations that are not cleanly interpretable via group or ring operations.
 ABLATIONS AND TRICKS We've tried various forms of regularization to see what can induce networks to generalize better on our datasets. Here we present the data efficiency curves on a particular dataset S 5 for a variety of interventions: full-batch gradient descent, stochastic gradient descent, large or small learning rates, residual dropout  (Srivastava et al., 2014) , weight decay  (Loshchilov & Hutter, 2017)  and gradient noise  (Neelakantan et al., 2015) . The results are shown in Figure  2  (left). We find that adding weight decay has a very large effect on data efficiency, more than halving the amount of samples needed compared to most other interventions. We found that weight decay towards the initialization of the network is also effective, but not quite as effective as weight decay towards the origin. This makes us believe that the prior, that approximately zero weights are suitable for small algorithmic tasks, explains part, but not all of the superior performance of weight decay. Adding some noise to the optimization process (e.g. gradient noise from using minibatches, Gaussian noise applied to weights before or after computing the gradients) is beneficial for generalization, consistent with the idea that such noise might induce the optimization to find flatter minima that generalize better. We found that learning rate had to be tuned in a relatively narrow window for the generalization to happen (within 1 order of magnitude).
 QUALITATIVE VISUALIZATION OF EMBEDDINGS In order to gain some insight into networks that generalize, we visualized the matrix of the output layer for the case of modular addition and S 5 . In Figure  3  we show t-SNE plots of the row vectors. For some networks we find clear reflections of the structure of the underlying mathematical objects in the plots. For example the circular topology of modular addition is shown with a 'number line' formed by adding 8 to each element. The structure is more apparent in networks that were optimized with weight decay.
 DISCUSSION We have seen that in the datasets we studied, small algorithmic binary operation tables, effects such as double descent or late generalization, and improvements to generalization from interventions like weight decay can be striking. This suggests that these datasets could be a good place to investigate aspects of generalization. For example, we plan to test whether various proposed measures of minima flatness correlate with generalization in our setting. We have also seen that visualizing the embedding spaces of these neural networks can show natural kinds of structure, for example in problems of modular arithmetic the topology of the embeddings tends to be circles or cylinders. We also see that the network tends to idiosyncratically organize the embeddings by various residues. Whilst the properties of these mathematical objects are familiar to us, we speculate that such visualizations could one day be a useful way to gain intuitions about novel mathematical objects. In addition, we document an interesting phenomenon, where the number of optimization steps needed to reach a given level of performance increases quickly as we reduce the size of the training dataset. Since this represents a way trade compute for performance on smaller amounts of data, it would be useful to investigate in future work whether the effect is also present for other datasets. The following are the binary operations that we have tried (for a prime number p = 97): x • y = x + y (mod p) for 0 ≤ x, y < p x • y = x − y (mod p) for 0 ≤ x, y < p x • y = x/y (mod p) for 0 ≤ x < p, 0 < y < p x • y = [x/y (mod p) if y is odd, otherwise x − y (mod p)] for 0 ≤ x, y < p x • y = x 2 + y 2 (mod p) for 0 ≤ x, y < p x • y = x 2 + xy + y 2 (mod p) for 0 ≤ x, y < p x • y = x 2 + xy + y 2 + x (mod p) for 0 ≤ x, y < p x • y = x 3 + xy (mod p) for 0 ≤ x, y < p x • y = x 3 + xy 2 + y (mod p) for 0 ≤ x, y < p x • y = x • y for x, y ∈ S 5 x • y = x • y • x −1 for x, y ∈ S 5 x • y = x • y • x for x, y ∈ S 5 For each binary operation we constructed a dataset of equations of the form x op y = x • y , where a stands for the token corresponding to element a. For each training run, we chose a fraction of all available equations at random and declared them to be the training set, with the rest of equations being the validation set.
 A.1.2 MODEL AND OPTIMIZATION We trained a standard decoder-only transformer  (Vaswani et al., 2017)  with causal attention masking, and calculated loss and accuracy only on the answer part of the equation. For all experiments we used a transformer with 2 layers, width 128, and 4 attention heads, with a total of about 4 • 10 5 non-embedding parameters. We have tuned optimization hyperparameters by running experiments on modular addition and product in S 5 . For final configuration of hyperparameters we have chosen a balance of performance we saw on S 5 and simplicity (for example we chose not to anneal the learning rate for the experiments in the paper even though it performed better in some situations). For most experiments we used AdamW optimizer with learning rate 10 −3 , weight decay 1, β 1 = 0.9, β 2 = 0.98, linear learning rate warmup over the first 10 updates, minibatch size 512 or half of training dataset size (whichever was smaller) and optimization budget of 10 5 gradient updates. In section 3.3 we have also tried the following variants (listed in the reading order for Figure  2 left ): • Adam optimizer with full batch (i.e. exact gradient of the loss on the whole training dataset) • Adam optimizer • Adam optimizer with full batch and Gaussian noise added to the update direction for each parameter (W ← W + lr • (∆W + ), where is sampled from unit Gaussian, ∆W is the standard Adam weight update, and lr is the learning rate) • Adam optimizer on model with residual dropout 0.1 added • AdamW optimizer with weight decay 1 (default setting in most other experiments) • AdamW optimizer with weight decay 1 towards the initialization instead of the origin • Adam optimizer with learning rate 3 • 10 −4 • Adam optimizer with learning rate 3 • 10 −3 • Adam optimizer on model with Gaussian weight noise of standard deviation 0.01 (i.e. each parameter W replaced by W + 0.01 • in the model, with sampled from unit Gaussian). For experiments reported in Section 3.1.1 we increased the optimization budget to 5 • 10 5 optimization steps in order to capture the increase of time to perfect generalization better. For the experiments reported in Section 3.1 we increased the optimization budget to 10 6 , and used Adam optimizer with no weight decay, for emphasizing how late into the optimization process the generalization can begin. We've repeated each experiment for each dataset size with 3 random seeds, with the exception of experiments in section 3.1.1, where we've aggregated results over 7 random seeds.
 A.2 ADDITIONAL FIGURES In Figure  4  we show the loss curves that correspond to the accuracy curves in Figure  1 . In Figure  5  we show an example of a binary operation table that the network can actually solve. Figure  4 : The loss curves for modular division, train and validation. We see the validation loss increases from 10 2 to about 10 5 optimization steps before it begins a second descent.
 A.3 RELATED WORK In this paper we study training and generalization dynamics on small simple algorithmic datasets. In the past, algorithmic datasets have been used to probe the capability of neural networks to perform symbolic and algorithmic reasoning. For example the tasks of copying, reversing, and sorting randomly generated sequences, and performing arithmetic operations of multi-digit numbers, have been used as standard benchmarks for sequence-to-sequence models  (Graves et al., 2014) ,  (Weston et al., 2014 ) )  (Reed & De Freitas, 2015) ,  (Grefenstette et al., 2015) ,  (Zaremba & Sutskever, 2015) ,  (Graves, 2016) ,  (Dehghani et al., 2018) . Typically in these works however the emphasis is on the performance in the unlimited data regime, with generalization often studied with respect to input sequence length. Some papers study the sample complexity on algorithmic tasks  (Reed & De Freitas, 2015) , but mostly focus on the impact of architectural choices. In contrast we study the phenomenon of generalization in data-limited regime, with an emphasis on phenomena that we believe to be architecture-agnostic. Algorithmically generated reasoning datasets like bAbI  (Weston et al., 2015)  encourage work on studying generalization in data-limited regime. Most results on such datasets however focus on a point estimate of performance of a particular architecture or training technique, whereas our main interest is in pointing out the change in generalization past the point where a particular architecture can memorize the training data completely.  (Neelakantan et al., 2015)  has a "grok-like" learning curve on an algorithmic task, but it is related to optimization difficulty, whereas our phenomenon is specifically about generalization. In  (Saxton et al., 2019)  they study generalization on procedurally generated math problems such as arithmetic and differentiation, but for the most part these tasks are more involved than the simple Figure  5 : One of the binary operation tables presented to the networks that the network can perfectly fill in. Each symbol is represented as a letter in English, Hebrew, or Greek alphabet for reader's convenience. We invite the reader to guess which operation is represented here. binary op problems we have studied and as such do not lend themselves to observing the kinds of phenomena we describe in this paper, since they would require an extremely large number of samples to master. In  (Jiang et al., 2019)  they studied a large number of generalization or complexity measures on convolutional neural networks to see which, if any, are predictive of generalization performance. They find that flatness based measures that aim to quantify the sensitivity of the trained neural network to parameter perturbations are the most predictive. We conjectured that the grokking phenomena we report in this work may be due to the noise from SGD driving the optimization to flatter/simpler solutions that generalize better and hope to investigate in future work whether any of these measures are predictive of grokking.  (Zhang et al., 2016)  finds that neural networks of sizes typically used in deep learning can interpolate arbitrary training data, and yet generalize when trained with semantically meaningful labels using appropriate optimization procedures. Our work shows a related phenomenon where neural networks can interpolate a small algorithmic training dataset without generalizing, but start generalizing when trained with SGD for longer. 1 st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021.  (Nakkiran et al., 2019; Belkin et al., 2018)  focus on the phenomenon of double descent in loss as a function of model and optimization procedure capacity. They find that the classical U-shaped validation loss curve is followed in some settings (including neural network training) by a second descent of loss that starts around the minimal capacity that is needed to interpolate any training data. We observe a second descent in validation loss (though not accuracy) as a function of the amount of training in some of our experiments, and it happens past the point of interpolating the training data. We believe that the phenomenon we describe might be distinct from the double descent phenomena described in  (Nakkiran et al., 2019; Belkin et al., 2018)  because we observe the second descent in loss far past the first time the training loss becomes very small (tens of thousands of epochs in some of our experiments), and we don't observe a non-monotonic behavior of accuracy. The setting of small algorithmic datasets that we study also provides a smaller, more tractable playground for studying subtle generalization phenomena than natural datasets studied in  (Nakkiran et al., 2019) . A.4 GENERALIZATION WITH MEMORIZING SEVERAL OUTLIERS In this situation one could imagine one of the following scenarios unfolding. If the model class of neural networks optimized and regularized as before was not large enough to interpolate such "noisy" dataset, one could imagine the procedure converging to a solution that generalizes well, but denoises the training data (i.e. predicts c = a • b as an answer even for the outlier equations a, b → c with c = c). On the other extreme it could be that the optimization procedure can find networks that interpolate the data, but the resulting models don't generalize, because they are forced to represent a considerably more complicated function than before (a simple function + k exceptions encoded in the training data). In our experiments we find that the first option doesn't happen -all experiments reach 100% training accuracy at some point, and this point is not considerably affected by changing the number of outliers k. The second phenomenon happens in a range of training data percentages and number of outliers k -increasing k decreases the range of training data percentages for which the optimization procedure converges to models that generalize. However the effect of introducing a small number of outliers (up to 1000) is not very pronounced -see Figure  6 . We interpret this as additional evidence that the capacity of the network and optimization procedure is well beyond the capacity needed for memorizing all the labels on the training data, and that generalization happening at all requires a non-trivial explanation. Figure 2 : 2 Figure 2: Left. Different optimization algorithms lead to different amounts of generalization within an optimization budget of 10 5 steps for the problem of learning the product in the abstract group S 5 .Weight decay improves generalization the most, but some generalization happens even with full batch optimizers and models without weight or activation noise at high percentages of training data. Suboptimal choice hyperparameters severely limit generalization. Not shown: training accuracy reaches 100% after 10 3 -10 4 updates for all optimization methods. Right. Best validation accuracy achieved after 10 5 steps on a variety of algorithmic datasets, averaged over 3 seeds. Generalization happens at higher percentages of data for intuitively more complicated and less symmetrical operations.
 Figure 3 : 3 Figure 3: Left. t-SNE projection of the output layer weights from a network trained on S 5 . We see clusters of permutations, and each cluster is a coset of the subgroup (0, 3)(1, 4), (1, 2)(3, 4) or one of its conjugates. Right. t-SNE projection of the output layer weights from a network trained on modular addition. The lines show the result of adding 8 to each element. The colors show the residue of each element modulo 8.
 Figure 6 : 6 Figure 6: Effect on data efficiency of introducing k ∈[0, 10, 100, 1000, 2000, 3000]  outliers (examples with random labels) into the training data. Small number of outliers doesn't noticeably impact generalization performance, but a large number hinders it significantly.

			  st  Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021.

Abstract: In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail.  In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.


Astronomical Waste: The Opportunity Cost of Delayed

2 being irreversibly degraded into entropy on a cosmic scale. These are resources that an advanced civilization could have used to create value-structures, such as sentient beings living worthwhile lives. The rate of this loss boggles the mind. One recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10 46 per century of delayed colonization.  1  This estimate assumes that all the lost entropy could have been used for productive purposes, although no currently known technological mechanisms are even remotely capable of doing that. Since the estimate is meant to be a lower bound, this radically unconservative assumption is undesirable. We can, however, get a lower bound more straightforwardly by simply counting the number or stars in our galactic supercluster and multiplying this number with the amount of computing power that the resources of each star could be used to generate using technologies for whose feasibility a strong case has already been made. We can then divide this total with the estimated amount of computing power needed to simulate one human life. As a rough approximation, let us say the Virgo Supercluster contains 10 13 stars. One estimate of the computing power extractable from a star and with an associated planet-sized computational structure, using advanced molecular nanotechnology 2 , is 10 42 operations per second. 3 A typical estimate of the human brain's processing power is roughly 10 17 operations per second or less. 4 Not much more seems to be needed to simulate the relevant parts of the environment in sufficient detail to enable the simulated minds to have experiences indistinguishable from typical current human experiences. 5 3 Given these estimates, it follows that the potential for approximately 10 38 human lives is lost every century that colonization of our local supercluster is delayed; or equivalently, about 10 29 potential human lives per second. While this estimate is conservative in that it assumes only computational mechanisms whose implementation has been at least outlined in the literature, it is useful to have an even more conservative estimate that does not assume a non-biological instantiation of the potential persons. Suppose that about 10 10 biological humans could be sustained around an average star. Then the Virgo Supercluster could contain 10 23 biological humans. This corresponds to a loss of potential of over 10 13 potential human lives per second of delayed colonization. What matters for present purposes is not the exact numbers but the fact that they are huge. Even with the most conservative estimate, assuming a biological implementation of all persons, the potential for over ten trillion potential human beings is lost for every second of postponement of colonization of our supercluster.  6
 II. THE OPPORTUNITY COST OF DELAYED COLONIZATION From a utilitarian perspective, this huge loss of potential human lives constitutes a correspondingly huge loss of potential value. I am assuming here that the human lives that could have been created would have been worthwhile ones. Since it is commonly supposed that even current human lives are typically worthwhile, this is a weak assumption. Any civilization advanced enough to colonize the local supercluster would likely also have the ability to establish at least the minimally favorable conditions required for future lives to be worth living.
 4 The effect on total value, then, seems greater for actions that accelerate technological development than for practically any other possible action. Advancing technology (or its enabling factors, such as economic productivity) even by such a tiny amount that it leads to colonization of the local supercluster just one second earlier than would otherwise have happened amounts to bringing about more than 10 29 human lives (or 10 13 human lives if we use the most conservative lower bound) that would not otherwise have existed. Few other philanthropic causes could hope to match that level of utilitarian payoff. Utilitarians are not the only ones who should strongly oppose astronomical waste. There are many views about what has value that would concur with the assessment that the current rate of wastage constitutes an enormous loss of potential value. For example, we can take a thicker conception of human welfare than commonly supposed by utilitarians (whether of a hedonistic, experientialist, or desire-satisfactionist bent), such as a conception that locates value also in human flourishing, meaningful relationships, noble character, individual expression, aesthetic appreciation, and so forth. So long as the evaluation function is aggregative (does not count one person's welfare for less just because there are many other persons in existence who also enjoy happy lives) and is not relativized to a particular point in time (no time-discounting), the conclusion will hold. These conditions can be relaxed further. Even if the welfare function is not perfectly aggregative (perhaps because one component of the good is diversity, the marginal rate of production of which might decline with increasing population size), it can still yield a similar bottom line provided only that at least some significant component of the good is sufficiently aggregative. Similarly, some degree of timediscounting future goods could be accommodated without changing the conclusion. 7
 III. THE CHIEF GOAL FOR UTILITARIANS SHOULD BE TO REDUCE EXISTENTIAL RISK In light of the above discussion, it may seem as if a utilitarian ought to focus her efforts on accelerating technological development. The payoff from even a very slight success in this endeavor is so enormous that it dwarfs that of almost any other activity. We appear to have a utilitarian argument for the greatest possible urgency of technological development. However, the true lesson is a different one. If what we are concerned with is (something like) maximizing the expected number of worthwhile lives that we will create, then in addition to the opportunity cost of delayed colonization, we have to take into account the risk of failure to colonize at all. We might fall victim to an existential risk, one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. 8 Because the lifespan of galaxies is measured in billions of years, whereas the time-scale of any delays that we could realistically affect would rather be measured in years or decades, the consideration of risk trumps the consideration of opportunity cost. For example, a single percentage point of reduction of existential risks would be worth (from a utilitarian expected utility point-ofview) a delay of over 10 million years. Therefore, if our actions have even the slightest effect on the probability of eventual colonization, this will outweigh their effect on when colonization takes place.
 6 For standard utilitarians, priority number one, two, three and four should consequently be to reduce existential risk. The utilitarian imperative 'Maximize expected aggregate utility!' can be simplified to the maxim 'Minimize existential risk!'.
 IV. IMPLICATIONS FOR AGGREGATIVE PERSON-AFFECTING VIEWS The argument above presupposes that our concern is to maximize the total amount of well-being. Suppose instead that we adopt a 'person-affecting' version of utilitarianism, according to which our obligations are primarily towards currently existing persons and to those persons that will come to exist. 9 On such a person-affecting view, human extinction would be bad only because it makes past or ongoing lives worse, not because it constitutes a loss of potential worthwhile lives. What ought someone who embraces this doctrine do? Should he emphasize speed or safety, or something else? To answer this, we need to consider some further matters. Suppose one thinks that the probability is negligible that any existing person will survive long enough to get to use a significant portion of the accessible astronomical resources, which, as described in opening section of this paper, are gradually going to waste. Then one's reason for minimizing existential risk is that sudden extinction would off cut an average of, say, 40 years from each of the current (six billion or so) human lives. 10 While this would certainly be a large disaster, it is in the same big ballpark as other ongoing human tragedies, such as world poverty, hunger and disease. On this assumption, then, a personaffecting utilitarian should regard reducing existential risk as a very important but not completely dominating concern. There would in this case be no easy answer to what he ought to do. Where he ought to focus his efforts would depend on detailed calculations about which area of philanthropic activity he would happen to be best placed to make a contribution to. Arguably, however, we ought to assign a non-negligible probability to some current people surviving long enough to reap the benefits of a cosmic diaspora. A socalled technological 'singularity' might occur in our natural lifetime 11 , or there could be a breakthrough in life-extension, brought about, perhaps, as result of machine-phase nanotechnology that would give us unprecedented control over the biochemical processes in our bodies and enable us to halt and reverse the aging process. 12 Many leading technologists and futurist thinkers give a fairly high probability to these developments happening within the next several decades.  13  Even if you are skeptical about their prognostications, you should consider the poor track record of technological forecasting. In view of the well-established unreliability of many such forecasts, it would seem unwarranted to be so confident in one's prediction that the requisite breakthroughs will not occur in our time as to give the hypothesis that they will a probability of less than, say, 1%. The expected utility of a 1% chance of realizing an astronomically large good could still be astronomical. But just how good would it be for (some substantial subset of) currently living people to get access to astronomical amounts of resources? The answer is not obvious. On the one hand, one might reflect that in today's world, the marginal utility for an individual of material resources declines quite rapidly ones his basic needs have been met. Bill Gate's level of well-being does not seem to dramatically exceed that of many a person of much more modest means. On the other hand, advanced technologies of the sorts that would most likely be deployed by the time we could colonize the local supercluster may well provide new ways of converting resources into well-being. In particular, material resources could be used to greatly expand our mental capacities and to indefinitely prolong our subjective lifespan. And it is by no means clear that the marginal utility of extended healthspan and increased mental powers must be sharply declining above some level. If there is no such decline in marginal utility, we have to conclude that the expected utility to current individuals 14 of successful colonization of our supercluster is astronomically great, and this conclusion holds even if one gives a fairly low probability to that outcome. A long shot it may be, but for an expected utility maximizer, the benefit of living for perhaps billions of subjective years with greatly expanded capacities under fantastically favorable conditions could more than make up for the remote prospects of success. Now, if these assumptions are made, what follows about how a person-affecting utilitarian should act? Clearly, avoiding existential calamities is important, not just because it would truncate the natural lifespan of six billion or so people, but also -and given the assumptions this is an even weightier consideration -because it would extinguish the chance that current people have of reaping the enormous benefits of eventual colonization. However, by contrast to the total utilitarian, the person-affecting utilitarian would have to balance this goal with another equally important desideratum, namely that of maximizing the chances of current people surviving to benefit from the colonization. For the person-affecting utilitarian, it is not enough that humankind survives to colonize; it is crucial that extant people be saved. This should lead her to emphasize speed of technological development, since the rapid arrival of advanced technology would surely be needed to help current people stay alive until the fruits of 9 colonization could be harvested. If the goal of speed conflicts with the goal of global safety, the total utilitarian should always opt to maximize safety, but the person-affecting utilitarian would have to balance the risk of people dying of old age with the risk of them succumbing in a species-destroying catastrophe. Mixed ethical views, which also incorporate non-utilitarian elements, might or might not yield one of these bottom lines depending on the nature of what is added. 15 1 M. Cirkovic, 'Cosmological Forecast and its Practical Significance', Journal of  Evolution and Technology, xii (2002) , http://www.jetpress.org/volume12/CosmologicalForecast.pdf. 10 7 Utilitarians commonly regard time-discounting as inappropriate in evaluating moral goods (see e.g. R. B.  Brandt, Morality, Utilitarianism, and Rights, Cambridge, 1992, pp. 23f.) . However, it is not clear that utilitarians can avoid compromising on this principle in view of the possibility that our actions could conceivably have consequences for an infinite number of persons (a possibility that we set aside for the purposes of this paper).  8 N. Bostrom, 'Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards', Journal of
			 K. E. Drexler, Nanosystems: Molecular Machinery, Manufacturing, and Computation, New York, John Wiley & Sons, Inc., 1992. 3 R. J. Bradbury, 'Matrioshka Brains', Manuscript, 2002, http://www.aeiveos.com/~bradbury/MatrioshkaBrains/MatrioshkaBrains.html 4 N. Bostrom, 'How Long Before Superintelligence?', International Journal of Futures Studies ii (1998); R. Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence, New York, Viking, 1999. The lower estimate is in H. Moravec, Robot: Mere Machine to Transcendent Mind, Oxford, 1999. 5 N. Bostrom, 'Are You Living in a Simulation?', Philosophical Quarterly, liii (211). See also http://www.simulation-argument.com.6  The Virgo Supercluster contains only a small part of the colonizable resources in the universe, but it is sufficiently big to make the point. The bigger the region we consider, the less certain we can be that significant parts of it will not have been colonized by a civilization of non-terrestrial origin by the time we could get there.
			 I'm grateful for comments from John Broome, Milan Cirkovic, Roger Crisp, Robin Hanson, and James Lenman, and for the financial support of a British Academy Postdoctoral Award.

Abstract:
With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for standard utilitarians is not that we ought to maximize the pace of technological development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a 'personaffecting' stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerations with other criteria, will also be committed to a similar bottom line.
 I. THE RATE OF LOSS OF POTENTIAL LIVES As I write these words, suns are illuminating and heating empty rooms, unused energy is being flushed down black holes, and our great common endowment of negentropy is Russian, Portuguese, Bosnian Translations:


Testing the automation revolution hypothesis ✩

Introduction Since at least 2013, many have claimed that we are entering an automation revolution, and so should soon expect large trenddeviating increases in job automation, in related job losses, and in the determinants of automation. As context for considering such claims, we study what predicts which jobs have been how automated in the recent past, how the best automation predictors have changed over time, and recent correlations between changes in automation, pay, and employment.
 Material We combine data from three sources  (Scholl and Hanson, 2020) . First, O*NET is a database of U.S. job feature scores made by surveying occupational experts and employees, who compare ✩ For financial support we thank Open Philanthropy Project for funding ''Research into Future Artificial Intelligence Scenarios.'' They did not influence our study design or interpretation. each job to related jobs. For each of 2144 jobs, O*NET includes 261 job features, many scored on a 0-5 numerical scale. In its current form O*NET started in 2002, though that year's entries include data from 1999. We project these scores onto 881 ''sixdigit'' level jobs, via scaling frequency by importance. We thus have scores of 261 job features for 881 jobs from 1999 to 2019. One key O*NET job feature is years of education, and another is ''degree of automation'', which seems to represent expert judgments on which workers would have to do each task now done by machines, absent those machines. Second, from Occupational Employment Statistics (OES), we obtain, for all these jobs and for years 1999 to 2018, U.S. annual averages for employment (i.e., number of employees) and an inflation-adjusted mean hourly pay (in U.S. dollars). Our third source is two widely-discussed expert-judgmentderived metrics regarding the vulnerability of jobs to near future automation. These metrics do not vary by year. Computerisable comes from  Frey and Osborne (2017) , first published in 2013, and ranges 0 to 1. Building on judgments made by a ''group of machine learning researchers'', they estimated 47% of U.S. jobs to be at ''high risk'' of being ''computerisable'', ''perhaps over the next decade or two''. Machine Learning Suitability, coming directly from an author of  Brynjolfsson et al. (2018) , was created from machine learning expert judgments using a 23-item rubric, and ranges 2 to 5. We transform all variables (besides time and intercept) into rough ''z-score'' variants. We apply a logarithmic transform to 0.01 plus pay, employment, and each O*NET variable. We then rescale them to have zero mean and unit standard deviation. Time is rescaled to take value zero at 1999, and one at 2019. Our transformed variables look closer to normally distributed. As we use transformed versions in our analysis, most regression coefficients say how many standard deviations of increase in a dependent variable is predicted by a one standard deviation increase in an independent variable. Most O*NET job features were not scored for each job every year, but were on average scored 3.3 times during our 1999-2019 period. From available scores, we interpolate scores for all years using two methods. In piecewise-linear interpolation, we fit straight lines in time between scorings that are adjacent in time, and fit zero-slope lines for years outside the time range of available scorings. In regression interpolation, we fit a linear regression in time to all available job-feature scorings. This method requires at least two job-feature scorings. While our dataset officially includes 881 jobs and 260 O*NET job features (besides education), data limitations force a tradeoff between how many jobs and features we can include. We choose to maximize the product of these at 832 jobs and 251 features. Table 3 further selects for jobs with enough data to allow regression interpolation for all features used.
 Methods Table  1  gives basic statistics for all variables in the other tables. Table  2  describes seven ordinary least squares regressions, all predicting automation. Each data point corresponds to a notinterpolated O*NET scoring of an automation value for a job in a year; independent variables are piecewise-linear interpolated. These 25 O*NET variables were selected via the LASSO method out of the 251 available O*NET variables via a structure like Model 3. Models 2,5 add extra columns for each variable multiplied by time, and Models 6,7 apply the same structure of Model 4 separately to times <0.5 and >0.5. A simple model not shown, using only an intercept and time, gives a time coefficient of 0.102, not significant at the 10% level. Table  3  describes regression models predicting changes in pay and employment. All change variables are constructed via regression interpolation. Changes are not renormalized into z-scores; they are differences in normalized z-scores. These models help us test these null hypotheses: (1) metrics constructed to forecast future automation do not predict past automation, (2) best predictors of automation have not changed in two decades, (3) automation changes do not predict changes in pay or employment.
 Results The first two models of Table  2  suggest that five variables, plus time and an intercept, have substantial predictive power, explaining roughly 15% of automation variation. Automation requires fixed costs, but saves on worker marginal costs. Simple theory thus predicts that, all else equal, employers are more eager to automate jobs with higher pay and employment. These two factors should thus predict job automation, and we do in fact see such effects, though more consistently for pay. Two automation vulnerability metrics built from expert judgments on which jobs seem easier to automate in the future predict which jobs were more automated in the last two decades. Education does not, after our other controls. Model 4 suggests that, aside perhaps from education, these predictors have little to add to the predictive power of the top 25 O*NET variables, which can explain over half of automation variance, even when interpolated. Our two strongest O*NET predictors are Pace Determined By Speed Of Equipment and Importance of Repeating Same Tasks, with coefficients of 0.58 and 0.23. Following these, we find a set of four predictors at roughly 0.14, and a set of five at roughly 0.08. Most of these predictors seem understandable in terms of traditional styles of job automation. For example, Pace Determined By Speed Of Equipment picks out jobs that coordinate closely with machinery, while Importance of Repeating Same Tasks picks out jobs with many similar and independent small tasks. The coefficients in Table  2  that probe time effects do not offer much support for the claim that automation predictors have changed noticeably over time. Model 5 finds only three of 31 time-interaction coefficients are 10% significant, the number to be expected at random. Models 6,7, show no clear differences between time periods. Though we expect automation to have increased over our period, we find no significant time coefficient using only an intercept and time, and adding many controls makes automation seem to have decreased with time. As scores come from expert judgments made at different times, this decrease may be due to drifting standards on what it takes for a job to seen as more automated  (Levari et al., 2018) . If so, the difference between these two time coefficients, roughly 0.37 standard deviations of automation, can be interpreted as estimating this much of an increase in the average suitability of jobs for automation over this period. In Table  3 , changes in pay or employment are only significantly predicted by changes in job automation in one model out of six, where the sign is the opposite of the usual fear. If real, that one coefficient confirms the basic theory that employers gain more from automating jobs with more workers. Changes in pay and employment consistently predict each other, and with large coefficients. This suggests that, in supply and demand terms, labor market changes are on average better seen as changes in demand, and less as changes in supply. Increases in education consistently predict declines in pay and employment, though with small coefficients, suggesting that falling labor demand has been positively correlated with increasing education. Perhaps labor shortages (surpluses) induce firms to adopt weaker (stricter) education requirements. Terms in Table  3  that interact changes in employment and pay with changes in automation suggest that the positive correlation between pay and employment changes is weaker for jobs that saw increased automation.
 Conclusions Recently, many have said we are entering an automation revolution which will soon produce large trend-deviating increases in automation levels and resulting job losses, and also big changes in which kinds of jobs are more vulnerable to automation. We do not yet see evidence of such a revolution. Using 1505 expert reports on the degree of automation of particular jobs 1999-2019, we find that reported automation levels have not changed noticeably, though changing reporting standards may mask such increases. Jobs with larger automation increases did not on average see noticeable changes in pay or employment. Both pay and employment predict automation in the direction suggested by simple theory, as do expert judgments on which jobs are vulnerable to future automation. We can explain over half the variance in which jobs have been how automated using the top 25 O*NET variables, which are relatively mundane and understandable in terms of traditional kinds of automation. All these best predictors have not noticeably changed in two decades. Table 1 1 Variable statistics. Variable Untransformed Transformed Mean Std Dev Min Max Min Max Time 2007.50 6.16 1999 2019 0 1 Education 14.31 2.60 10.28 22.88 -1.839 2.824 Employees 167538 387560 200 4612510 -3.428 2.857 Pay 24.67 13.94 7.18 129.62 -2.331 3.735 Machine Learning Suitability 3.466 0.115 2.780 3.902 -6.609 3.586 Computerisable 0.536 0.368 0.003 0.990 -2.284 0.845 O*NET: Activity 3.374 0.397 1.750 4.620 -5.297 2.629 Advancement 2.723 0.423 1.250 4.000 -4.701 2.444 Cramped 1.932 0.764 1.000 4.900 -1.562 2.680 Dynamic Strength 0.079 0.086 0.000 0.736 -1.389 2.303 Fine Arts 0.040 0.110 0.000 0.974 -0.859 3.784 Gross Body Equilibrium 0.060 0.072 0.000 0.574 -1.326 2.450 Hearing Sensitivity 0.125 0.090 0.000 0.904 -3.253 2.944 Importance of Repeating Same Tasks 2.859 0.846 1.100 5.000 -2.801 1.883 Indoors Environmentally Controlled 3.998 0.945 1.000 5.000 -4.356 0.853 Innovation 3.552 0.489 1.880 4.880 -4.345 2.276 Letters and Memos 3.264 0.791 1.090 5.000 -3.894 1.693 Mathematics 0.264 0.164 0.000 0.989 -4.648 2.218 Number Facility 0.199 0.117 0.000 0.800 -4.712 2.491 On Knees 1.982 0.689 1.000 4.660 -2.205 2.794 Pace Determined by Speed of Equipment 1.839 0.826 1.000 4.760 -1.303 2.588 Physical Proximity 3.451 0.688 1.290 5.000 -4.729 1.923 Spend Time Keeping or Regaining Balance 1.570 0.525 1.000 4.310 -1.321 3.486 Spend Time Sitting 3.126 0.975 1.010 5.000 -3.112 1.523 Supervision Human Relations 3.200 0.467 1.250 4.620 -5.539 2.274 Supervision Technical 2.781 0.585 1.120 4.620 -3.730 2.256 Support 3.778 0.979 1.250 7.000 -5.753 2.618 Thinking Creatively 0.312 0.206 0.000 0.928 -3.456 1.507 Variety 2.791 0.637 1.120 4.120 -3.583 1.699 Visualization 0.221 0.113 0.000 0.657 -5.038 2.028 Wear Common Safety Equipment 2.736 1.346 1.000 5.000 -1.578 1.355
 Table 2 2 Predicting Automation. Model (1) (2) (3) (4) (5) (6) (7) Intercept 0.2146*** 0.3397*** 0.1474*** 0.1755*** 0.1939* 0.2809*** -0.0097 (0.0550) (0.1043) (0.0435) (0.0456) (0.1011) (0.0828) (0.1545) Time -0.2982*** -1.0041** 0.7372** -0.2451*** -0.2682*** -0.5104 0.2796 -0.6632*** -0.0351 (0.0944) (0.3948) (0.3538) (0.0849) (0.0857) (0.4035) (0.3918) (0.2386) (0.1984) Education 0.0099 -0.1101 0.2452 0.0906** 0.0370 0.0630 0.0938 0.0681 (0.0444) (0.1114) (0.1925) (0.0453) (0.1093) (0.1929) (0.0657) (0.0637) Employees 0.0893*** 0.0422 0.0883 0.0362* -0.0024 0.0611 0.0339 0.0296 (0.0248) (0.0593) (0.1056) (0.0201) (0.0496) (0.0865) (0.0295) (0.0288) Pay 0.2286*** 0.2734*** -0.0937 0.0508 0.0527 -0.0267 0.0586 0.0206 (0.0369) (0.0905) (0.1555) (0.0342) (0.0856) (0.1445) (0.0518) (0.0464) Computerisable 0.3356*** 0.3771*** -0.0820 0.0048 0.0067 0.0069 0.0012 0.0232 (0.0312) (0.0754) (0.1326) (0.0276) (0.0679) (0.1180) (0.0411) (0.0368) M.L. Suitability 0.2161*** 0.2786*** -0.1212 0.0006 0.0273 -0.0463 0.0188 -0.0181 (0.0246) (0.0581) (0.1005) (0.0202) (0.0482) (0.0825) (0.0289) (0.0281) Activity 0.0336 0.0142 -0.0448 0.1116 -0.0082 0.0340 (0.0210) (0.0220) (0.0527) (0.0938) (0.0320) (0.0301) Advancement 0.0656*** 0.0551** 0.1036* -0.0991 0.0605 0.0346 (0.0250) (0.0254) (0.0618) (0.1086) (0.0370) (0.0346) Cramped -0.0527 -0.0594 -0.1391 0.1811 -0.1138* 0.0161 (0.0393) (0.0399) (0.0975) (0.1672) (0.0592) (0.0542) Dynamic Strength -0.0663 -0.0552 -0.0060 -0.1442 -0.0420 -0.1199* (0.0505) (0.0508) (0.1228) (0.2159) (0.0751) (0.0706) Fine Arts -0.0463* -0.0358 -0.0932 0.1117 -0.0706* -0.0157 (0.0267) (0.0270) (0.0637) (0.1108) (0.0401) (0.0368) Gross Body Equilibrium 0.0036 -0.0024 0.0260 -0.0247 0.0269 0.0112 (0.0489) (0.0489) (0.1182) (0.2042) (0.0732) (0.0664) Hearing Sensitivity -0.0713** -0.0757*** -0.0311 -0.0857 -0.0560 -0.0966** (0.0291) (0.0291) (0.0615) (0.1218) (0.0383) (0.0477) Importance of Repeating Same Tasks 0.2240*** 0.2254*** 0.1513** 0.1302 0.1975*** 0.2587*** (0.0317) (0.0326) (0.0770) (0.1458) (0.0449) (0.0546) Indoors Environmentally Controlled 0.1375*** 0.1357*** 0.1491*** -0.0382 0.1358*** 0.1316*** (0.0242) (0.0245) (0.0553) (0.1003) (0.0330) (0.0376) Innovation -0.0887*** -0.0924*** -0.0723 -0.0407 -0.0919*** -0.0938** (0.0248) (0.0251) (0.0545) (0.0999) (0.0346) (0.0385) Letters and Memos 0.1624*** 0.1457*** 0.0748 0.1467 0.1185*** 0.1882*** (0.0273) (0.0281) (0.0653) (0.1151) (0.0399) (0.0408) Mathematics 0.0979** 0.0838** 0.0770 0.0699 0.0734 0.1610** (0.0407) (0.0412) (0.0968) (0.1893) (0.0569) (0.0661) Number Facility 0.0415 0.0400 -0.0151 0.0795 0.0177 0.0141 (0.0359) (0.0359) (0.0794) (0.1679) (0.0467) (0.0624) On Knees -0.0462 -0.0303 -0.0072 -0.0695 0.0150 -0.0991* (0.0418) (0.0427) (0.0997) (0.1721) (0.0632) (0.0581) Pace Determined by Speed of Equipment 0.5638*** 0.5805*** 0.6974*** -0.2315* 0.6360*** 0.5221*** (0.0288) (0.0295) (0.0726) (0.1223) (0.0444) (0.0397) Physical Proximity -0.0694*** -0.0735*** -0.0440 -0.0496 -0.0755** -0.0641* (0.0217) (0.0221) (0.0508) (0.0912) (0.0301) (0.0330) Spend Time Keeping or Regaining Balance -0.0480 -0.0492 -0.0531 0.0049 -0.0599 -0.0368 (0.0395) (0.0400) (0.0953) (0.1632) (0.0587) (0.0546) Spend Time Sitting 0.0545* 0.0428 0.1454* -0.2272* 0.0472 0.0122 (0.0304) (0.0310) (0.0742) (0.1270) (0.0460) (0.0430) Supervision Human Relations -0.0117 -0.0025 -0.0562 0.0714 -0.0059 -0.0156 (0.0318) (0.0322) (0.0895) (0.1505) (0.0542) (0.0424) Supervision Technical 0.0607* 0.1068*** -0.0361 0.2352 0.0554 0.1366** (0.0350) (0.0400) (0.1013) (0.1734) (0.0617) (0.0539) Support 0.0539* 0.0363 0.2297** -0.2896* 0.1004 0.0055 (0.0314) (0.0320) (0.1067) (0.1557) (0.0725) (0.0361) Thinking Creatively -0.1375*** -0.1543*** -0.1222 0.0477 -0.1019* -0.0987 (0.0436) (0.0447) (0.0957) (0.1922) (0.0596) (0.0884) Variety -0.0647** -0.0697** -0.0950 0.0547 -0.0656 -0.0596 (0.0309) (0.0311) (0.0737) (0.1281) (0.0442) (0.0439) Visualization -0.0190 -0.0160 0.0597 -0.2230 -0.0246 -0.0508 (0.0319) (0.0322) (0.0686) (0.1395) (0.0423) (0.0534) Wear Common Safety Equipment -0.1384*** -0.1470*** -0.1847** 0.0813 -0.1579*** -0.1352*** (0.0329) (0.0333) (0.0831) (0.1426) (0.0508) (0.0444) N 1505 1505 1505 1505 1505 821 684 R2 Adjusted 0.1481 0.1509 0.5392 0.5407 0.5441 0.5288 0.5595 R2 0.1515 0.1576 0.5471 0.5501 0.5628 0.5466 0.5795 Dependent variable is Automation (A); *p < .1, **p < .05, ***p < .01, Standard errors in parentheses.
 Table 3 3 Predicting △ Pay, △ Employees. Model (1) (2) (3) (4) (5) (6) Dependent Variable: △ Pay △ Pay △ Pay △ Empl. △ Empl. △ Empl. Intercept 0.376*** 0.387*** 0.386*** -0.021** -0.002 -0.088*** (0.008) (0.011) (0.010) (0.008) (0.012) (0.022) △ A -0.006 -0.007 -0.009 0.001 0.000 0.041*** (0.005) (0.006) (0.006) (0.006) (0.006) (0.015) △ A * A(0) 0.013 0.011 0.009 0.008 (0.008) (0.008) (0.009) (0.009) △ A * △ A 0.006 0.007* 0.003 0.003 (0.004) (0.004) (0.004) (0.004) △ Education -0.015** -0.011* -0.021*** -0.016** (0.006) (0.006) (0.007) (0.007) △ Employees 0.172*** (0.040) △ A * △ Employees -0.078*** (0.028) △ Pay 0.216*** (0.048) △ A * △ Pay -0.104*** (0.035) N 495 495 495 495 495 495 R2 Adjusted 0.0004 0.0092 0.0586 -0.002 0.0115 0.0626 R2 0.0024 0.0172 0.0701 0.000 0.0195 0.0739 A = Automation. Standard errors in parentheses. * p<.1, ** p<.05, ***p<.01

Abstract: Wages and employment predict automation in 832 U.S. jobs, 1999 to 2019, but add little to top 25 O*NET job features, whose best predictive model did not change over this period. Automation changes predict changes in neither wages nor employment.