What do coherence arguments imply about the behavior of advanced AI?

Published 8 April 2021
This is an initial page, in the process of review, which may not be comprehensive or represent the best available understanding.
Coherence arguments say that if an entity's preferences do not adhere to the axioms of expected utility theory, then that entity is susceptible to losing things that it values.
This does not imply that advanced AI systems must adhere to these axioms ('be coherent'), or that they must be goal-directed.
Such arguments do appear to suggest that there will be non-zero pressure for advanced AI to become more coherent, and arguably also more 'goal-directed', given some minimal initial level of goal-directedness.
Details
Motion toward coherence
Expected utility maximization
'Maximizing expected utility' is a decision-making strategy, in which you assign a value to each possible 'outcome', and assign a probability to each outcome conditional on each of your available actions, then always choose the action whose resulting outcomes have the highest 'expected value' (average value of outcomes, weighted by the probability of those outcomes).
Coherence arguments
'Coherence arguments'1 demonstrate that if one's preferences cannot be understood as 'maximizing expected utility', then one can be manipulated into giving up things that one values for no gain.
For instance, one coherence argument notes that if you have 'circular preferences' then you will consent to series of decisions that will leave you worse off, given your preferences:
Suppose you prefer:
apple over pearpear over quincequince over appleany fruit over nothing
Then there is some tiny amount of money you would pay to go from apple to quince, and quince to pear, and pear to apple. At which point, you have spent money and are back where you started. If it is also possible to buy all of these fruit for money, then losing money means you lost some of whatever fruit for nothing, and you do want all of the fruit, by assumption.
If you avoid all such predictable losses, then according to the coherence arguments, you must be maximizing expected utility (as described above).
Coherence forces
That a certain characteristic of an entity's 'preferences' makes it vulnerable to manipulation does not mean that it will not have that characteristic. In order for such considerations to change the nature of an entity, ignoring outside intervention, something like the following conditions need to hold:
The entity can detect the characteristic (which could be difficult, if it is a logical relationship between all of its 'preferences' which are perhaps not straightforwardly accessible or well-defined)The realistic chance for loss is large enough to cause the entity to prioritize the problemThe entity is motivated to become coherent by the possibility of loss (versus for instance inferring that losing money is good, since it is equivalent to a set of exchanges that are each good)The entity is in a position to alter its own preferences
Similar might apply if versions of the above hold for an outside entity with power over the agent, e.g. its creators, though in that case it is less clear that 'coherence' is a further motivator beyond that for having the agent's preferences align with those of the outside agent (which would presumably coincide with coherence, to the extant that the outside agent had more coherent preferences).
Thus we say there is generally an incentive for coherence, but it may or may not actually cause a an entity to change in the direction of coherence at a particular time. We can also describe this as a 'coherence force' or 'coherence pressure', pushing minds toward coherence, all things equal, but for all we know, so weakly as to be often irrelevant.
Coherence forces apply to entities with 'preferences'
The coherence arguments only apply to creatures with 'preferences' that might be thwarted by their choices, so there are presumably possible entities that are not subject to any coherence forces, due to not having preferences of the relevant type.
Behavior of coherent creatures
Supposing entities are likely to become more coherent, all things equal, a natural question is how coherent entities differ from incoherent entities.
Coherence is consistent with any behavior
If we observe an agent exhibiting any history of behavior, that is consistent with the agent's being coherent because the agent could have a utility function that rates that history higher than any other history. Rohin Shah discusses this.
Coherence and goal-directedness
Coherence doesn't logically require goal-directedness
As Rohin Shah discusses, the above means that coherence does not imply 'goal-directed' behavior (however you choose to define that, if it doesn't include all behavior):
Coherence arguments do not exclude any behaviorNon-goal-directed behavior is consistent with coherence argumentsThus coherence arguments do not imply goal directed behavior
Increasing coherence seems likely to be associated with increased intuitive 'goal-directedness'
The following hypotheses (quoted from this blog post) seem plausible (where goal-directednessRohin means something like 'that which looks intuitively goal-directed)2:
1. Coherence-reformed entities will tend to end up looking similar to their starting point but less conflictedFor instance, if a creature starts out being indifferent to buying red balls when they cost between ten and fifteen blue balls, it is more likely to end up treating red balls as exactly 12x the value of blue balls than it is to end up very much wanting the sequence where it takes the blue ball option, then the red ball option, then blue, red, red, blue, red. Or wanting red squares. Or wanting to ride a dolphin.[…]2. More coherent strategies are systematically less wasteful, and waste inhibits goal-directionRohin, which means more coherent strategies are more forcefully goal-directedRohin on averageIn general, if you are sometimes a force for A and sometimes a force against A, then you are not moving the world with respect to A as forcefully as you would be if you picked one or the other. Two people intermittently changing who is in the driving seat, who want to go to different places, will not cover distance in any direction as effectively as either one of them. A company that cycles through three CEOs with different evaluations of everything will—even if they don't actively scheme to thwart one another—tend to waste a lot of effort bringing in and out different policies and efforts (e.g. one week trying to expand into textiles, the next week trying to cut everything not involved in the central business).3. Combining points 1 and 2 above, as entities become more coherent, they generally become more goal-directedRohin. As opposed to, for instance, becoming more goal-directedRohin on average, but individual agents being about as likely to become worse as better as they are reformed. Consider: a creature that values red balls at 12x blue balls is very similar to one that values them inconsistently, except a little less wasteful. So it is probably similar but more goal-directedRohin. Whereas it's fairly unclear how goal-directedRohin a creature that wants to ride a dolphin is compared to one that wanted red balls inconsistently much. In a world with lots of balls and no possible access to dolphins, it might be much less goal-directedRohin, in spite of its greater coherence. 4. Coherence-increasing processes rarely lead to non-goal-directedRohin agents—like the one that twitches on the ground In the abstract, few starting points and coherence-motivated reform processes will lead to an agent with the goal of carrying out a specific convoluted moment-indexed policy without regard for consequence, like Rohin's twitching agent, or to valuing the sequence of history-action pairs that will happen anyway, or to being indifferent to everything. And these outcomes will be even less likely in practice, where AI systems with anything like preferences probably start out caring about much more normal things, such as money and points and clicks, so will probably land at a more consistent and shrewd version of that, if 1 is true. (Which is not to say that you couldn't intentionally create such a creature.)
Thus it presently seems likely that coherence arguments correspond to a force for for entities with something like 'preferences' to grow increasingly coherent, and generally increasingly goal-directed (intuitively defined).
Thus, to the extent that future advanced AI has preferences of the relevant kind, there appears to be a pressure for it to become more goal-directed. However it is unclear what can be said generally about the strength of this force.
Primary author: Katja Grace
Notes


April files

By Katja Grace, April 1 2021
Today we are sharing with our blog readers a collection of yet-to-be-published drafts, in the hope of receiving feedback. We are especially looking for methodological critique, but all comments welcome!
Human-level performance estimate (Katja Grace)
How much hardware will we need to create AGI? (Asya Bergal, original idea credit to Ronny Fernandez)
Historic trends in AI Impacts productivity (Daniel Kokotajlo)
Analysis of superbombs as a global threat (Katja Grace)

April files


Coherence arguments imply a force for goal-directed behavior

By Katja Grace, Mar 25 2021
[Epistemic status: my current view, but I haven't read all the stuff on this topic even in the LessWrong community, let alone more broadly.]
There is a line of thought that says that advanced AI will tend to be 'goal-directed'—that is, consistently doing whatever makes certain favored outcomes more likely—and that this is to do with the 'coherence arguments'. Rohin Shah, and probably others1, have argued against this. I want to argue against them.
The old argument for coherence implying (worrisome) goal-directedness
I'd reconstruct the original argument that Rohin is arguing against as something like this (making no claim about my own beliefs here):
'Whatever things you care about, you are best off assigning consistent numerical values to them and maximizing the expected sum of those values' 'Coherence arguments2' mean that if you don't maximize 'expected utility' (EU)—that is, if you don't make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you'll be vulnerable to 'money-pumping'—being predictably parted from your money for nothing.3'Advanced AI will tend to do better things instead of worse things, by its own lights'Advanced AI will tend to avoid options that are predictably strictly worse by its own lights, due to being highly optimized for making good choices (by some combination of external processes that produced it, its own efforts, and the selection pressure acting on its existence).'Therefore advanced AI will maximize EU, roughly'Advanced AI will tend to be fairly coherent, at least to a level of approximation where becoming more coherent isn't worth the cost.4 Which will probably be fairly coherent (e.g. close enough to coherent that humans can't anticipate the inconsistencies).'Maximizing EU is pretty much the same as being goal-directed'To maximize expected utility is to pursue the goal of that which you have assigned higher utility to.5
And since the point of all this is to argue that advanced AI might be hard to deal with, note that we can get to that conclusion with:
'Highly intelligent goal-directed agents are dangerous'If AI systems exist that very competently pursue goals, they will likely be better than us at attaining their goals, and therefore to the extent there is a risk of mismatch between their goals and ours, we face a serious risk.
Rohin's counterargument
Rohin's counterargument begins with an observation made by others before: any behavior is consistent with maximizing expected utility, given some utility function. For instance, a creature just twitching around on the ground may have the utility function that returns 1 if the agent does whatever it in fact does in each situation (where 'situation' means, 'entire history of the world so far'), and 0 otherwise. This is a creature that just wants to make the right twitch in each detailed, history-indexed situation, with no regard for further consequences. Alternately the twitching agent might care about outcomes, but just happen to want the particular holistic unfolding of the universe that is occurring, including this particular series of twitches. Or it could be indifferent between all outcomes.
The basic point is that rationality doesn't say what 'things' you can want. And in particular, it doesn't say that you have to care about particular atomic units that larger situations can be broken down into. If I try to call you out for first spending money to get to Paris, then spending money to get back from Paris, there is nothing to say you can't just have wanted to go to Paris for a bit and then to come home. In fact, this is a common human situation. 'Aha, I money pumped you!' says the airline, but you aren't worried. The twitching agent might always be like this—a creature of more refined tastes, who cares about whole delicate histories and relationships, rather than just summing up modular momentarily-defined successes. And given this freedom, any behavior might conceivably be what a creature wants.
Then I would put the full argument, as I understand it, like this:
Any observable sequence of behavior is consistent with the entity doing EU maximization (see observation above)Doing EU maximization doesn't imply anything about what behavior we might observe (from 1)In particular, knowing that a creature is an EU maximizer doesn't imply that it will behave in a 'goal-directed' way, assuming that that concept doesn't apply to all behavior. (from 2)
Is this just some disagreement about the meaning of the word 'goal-directed'? No, because we can get back to a major difference in physical expectations by adding:
 Not all behavior in a creature implicates dire risk to humanity, so any concept of goal-directedness that is consistent with any behavior—and so might be implied by the coherence arguments—cannot imply AI risk.
So where the original argument says that the coherence arguments plus some other assumptions imply danger from AI, this counterargument says that they do not.
(There is also at least some variety in the meaning of 'goal-directed'. I'll use goal-directedRohin to refer to what I think is Rohin's preferred usage: roughly, that which seems intuitively goal directed to us, e.g. behaving similarly across situations, and accruing resources, and not flopping around in possible pursuit of some exact history of personal floppage, or peaceably preferring to always take the option labeled 'A'.6)
My counter-counterarguments
What's wrong with Rohin's counterargument? It sounded tight.
In brief, I see two problems:
The whole argument is in terms of logical implication. But what seems to matter is changes in probability. Coherence doesn't need to rule out any behavior to matter, it just has to change the probabilities of behaviors. Understood in terms of probability, argument 2 is a false inference: just because any sequence of behavior is consistent with EU maximization doesn't mean that EU maximization says nothing about what behavior we will see, probabilistically. All it says is that the probability of a behavioral sequence is never reduced to zero by considerations of coherence alone, which is hardly saying anything.
You might then think that a probabilistic version still applies: since every entity appears to be in good standing with the coherence arguments, the arguments don't exert any force, probabilistically, on what entities we might see. But:
An outside observer being able to rationalize a sequence of observed behavior as coherent doesn't mean that the behavior is actually coherent. Coherence arguments constrain combinations of external behavior and internal features—'preferences'7 and beliefs. So whether an actor is coherent depends on what preferences and beliefs it actually has. And if it isn't coherent in light of these, then coherence pressures will apply, whether or not its behavior looks coherent. And in many cases, revision of preferences due to coherence pressures will end up affecting external behavior. So 2) is not only not a sound inference from 1), but actually a wrong conclusion: if a system moves toward EU maximization, that does imply things about the behavior that we will observe (probabilistically).
Perhaps Rohin only meant to argue about whether it is logically possible to be coherent and not goal-directed-seeming, for the purpose of arguing that humanity can construct creatures in that perhaps-unlikely-in-nature corner of mindspace, if we try hard. In which case, I agree that it is logically possible. But I think his argument is often taken to be relevant more broadly, to questions of whether advanced AI will tend to be goal-directed, or to be goal-directed in places where they were not intended to be.
I take 1) to be fairly clear. I'll lay out 2) in more detail.
My counter-counterarguments in more detail
How might coherence arguments affect creatures?
Let us step back.
How would coherence arguments affect an AI system—or anyone—anyway? They're not going to fly in from the platonic realm and reshape irrational creatures.
The main routes, as I see it, are via implying:
incentives for the agent itself to reform incoherent preferencesincentives for the processes giving rise to the agent (explicit design, or selection procedures directed at success) to make them more coherentsome advantage for coherent agents in competition with incoherent agents
To be clear, the agent, the makers, or the world are not necessarily thinking about the arguments here—the arguments correspond to incentives in the world, which these parties are responding to. So I'll often talk about 'incentives for coherence' or 'forces for coherence' rather than 'coherence arguments'.
I'll talk more about 1 for simplicity, expecting 2 and 3 to be similar, though I haven't thought them through.
Looking coherent isn't enough: if you aren't coherent inside, coherence forces apply
If self-adjustment is the mechanism for the coherence, this doesn't depend on what a sequence of actions looks like from the outside, but from what it looks like from the inside.
Consider the aforementioned creature just twitching sporadically on the ground. Let's call it Alex.
As noted earlier, there is a utility function under which Alex is maximizing expected utility: the one that assigns utility 1 to however Alex in fact acts in every specific history, and utility 0 to anything else.
But from the inside, this creature you excuse as 'maybe just wanting that series of twitches' has—let us suppose—actual preferences and beliefs. And if its preferences do not in fact prioritize this elaborate sequence of twitching in an unconflicted way, and it has the self-awareness and means to make corrections, then it will make corrections8. And having done so, its behavior will change.
Thus excusable-as-coherent Alex is still moved by coherence arguments, even while the arguments have no complaints about its behavior per se.
For a more realistic example: suppose Assistant-Bot is observed making this sequence of actions:
Offers to buy gym membership for $5/week Consents to upgrade to gym-pro membership for $7/week, which is like gym membership but with added morning classesTakes discounted 'off-time' deal, saving $1 per week for only using gym in evenings
This is consistent with coherence: Assistant-Bot might prefer that exact sequence of actions over all others, or might prefer incurring gym costs with a larger sum of prime factors, or might prefer talking to Gym-sales-bot over ending the conversation, or prefer agreeing to things.
But suppose that in fact, in terms of the structure of the internal motivations producing this behavior, Assistant-Bot just prefers you to have a gym membership, and prefers you to have a better membership, and prefers you to have money, but is treating these preferences with inconsistent levels of strength in the different comparisons. Then there appears to be a coherence-related force for Assistant-Bot to change. One way that that could look is that since Assistant-Bot's overall behavioral policy currently entails giving away money for nothing, and also Assistant-Bot prefers money over nothing, that preference gives Assistant-Bot reason to alter its current overall policy, to avert the ongoing exchange of money for nothing.9 And if its behavioral policy is arising from something like preferences, then the natural way to alter it is via altering those preferences, and in particular, altering them in the direction of coherence.
One issue with this line of thought is that it's not obvious in what sense there is anything inside a creature that corresponds to 'preferences'. Often when people posit preferences, the preferences are defined in terms of behavior. Does it make sense to discuss different possible 'internal' preferences, distinct from behavior? I find it helpful to consider the behavior and 'preferences' of groups:
Suppose two cars are parked in driveways, each containing a couple. One couple are just enjoying hanging out in the car. The other couple are dealing with a conflict: one wants to climb a mountain together, and the other wants to swim in the sea together, and they aren't moving because neither is willing to let the outing proceed as the other wants. 'Behaviorally', both cars are the same: stopped. But their internal parts (the partners) are importantly different. And in the long run, we expect different behavior: the car with the unconflicted couple will probably stay where it is, and the conflicted car will (hopefully) eventually resolve the conflict and drive off.
I think here it makes sense to talk about internal parts, separate from behavior, and real. And similarly in the single agent case: there are physical mechanisms producing the behavior, which can have different characteristics, and which in particular can be 'in conflict'—in a way that motivates change—or not. I think it is also worth observing that humans find their preferences 'in conflict' and try to resolve them, which is suggests that they at least are better understood in terms of both behavior and underlying preferences that are separate from it.
So we have: even if you can excuse any seizuring as consistent with coherence, coherence incentives still exert a force on creatures that are in fact incoherent, given their real internal state (or would be incoherent if created). At least if they or their creator have machinery for noticing their incoherence, caring about it, and making changes.
Or put another way, coherence doesn't exclude overt behaviors alone, but does exclude combinations of preferences, and preferences beget behaviors. This changes how specific creatures behave, even if it doesn't entirely rule out any behavior ever being correct for some creature, somewhere.
That is, the coherence theorems may change what behavior is likely to appear amongst creatures with preferences.
Reform for coherence probably makes a thing more goal-directedRohin
Ok, but moving toward coherence might sound totally innocuous, since, per Rohin's argument, coherence includes all sorts of things, such as absolutely any sequence of behavior.
But the relevant question is again whether a coherence-increasing reform process is likely to result in some kinds of behavior over others, probabilistically.
This is partly a practical question—what kind of reform process is it? Where a creature ends up depends not just on what it incoherently 'prefers', but on what kinds of things its so-called 'preferences' are at all10, and what mechanisms detect problems, and how problems are resolved.
My guess is that there are also things we can say in general. It's is too big a topic to investigate properly here, but some initially plausible hypotheses about a wide range of coherence-reform processes:
Coherence-reformed entities will tend to end up looking similar to their starting point but less conflictedFor instance, if a creature starts out being indifferent to buying red balls when they cost between ten and fifteen blue balls, it is more likely to end up treating red balls as exactly 12x the value of blue balls than it is to end up very much wanting the sequence where it takes the blue ball option, then the red ball option, then blue, red, red, blue, red. Or wanting red squares. Or wanting to ride a dolphin.(I agree that if a creature starts out valuing Tuesday-red balls at fifteen blue balls and yet all other red balls at ten blue balls, then it faces no obvious pressure from within to become 'coherent', since it is not incoherent.)More coherent strategies are systematically less wasteful, and waste inhibits goal-directionRohin, which means more coherent strategies are more forcefully goal-directedRohin on averageIn general, if you are sometimes a force for A and sometimes a force against A, then you are not moving the world with respect to A as forcefully as you would be if you picked one or the other. Two people intermittently changing who is in the driving seat, who want to go to different places, will not cover distance in any direction as effectively as either one of them. A company that cycles through three CEOs with different evaluations of everything will—even if they don't actively scheme to thwart one another—tend to waste a lot of effort bringing in and out different policies and efforts (e.g. one week trying to expand into textiles, the next week trying to cut everything not involved in the central business).
Combining points 1 and 2 above, as entities become more coherent, they generally become more goal-directedRohin. As opposed to, for instance, becoming more goal-directedRohin on average, but individual agents being about as likely to become worse as better as they are reformed. Consider: a creature that values red balls at 12x blue balls is very similar to one that values them inconsistently, except a little less wasteful. So it is probably similar but more goal-directedRohin. Whereas it's fairly unclear how goal-directedRohin a creature that wants to ride a dolphin is compared to one that wanted red balls inconsistently much. In a world with lots of balls and no possible access to dolphins, it might be much less goal-directedRohin, in spite of its greater coherence.
Coherence-increasing processes rarely lead to non-goal-directedRohin agents—like the one that twitches on the groundIn the abstract, few starting points and coherence-motivated reform processes will lead to an agent with the goal of carrying out a specific convoluted moment-indexed policy without regard for consequence, like Rohin's twitching agent, or to valuing the sequence of history-action pairs that will happen anyway, or to being indifferent to everything. And these outcomes will be even less likely in practice, where AI systems with anything like preferences probably start out caring about much more normal things, such as money and points and clicks, so will probably land at a more consistent and shrewd version of that, if 1 is true. (Which is not to say that you couldn't intentionally create such a creature.)
These hypotheses suggest to me that the changes in behavior brought about by coherence forces favor moving toward goal-directednessRohin, and therefore at least weakly toward risk.
Does this mean advanced AI will be goal-directedRohin?
Together, this does not imply that advanced AI will tend to be goal-directedRohin. We don't know how strong such forces are. Evidently not so strong that humans11, or our other artifacts, are whipped into coherence in mere hundreds of thousands of years12. If a creature doesn't have anything like preferences (beyond a tendency to behave certain ways), then coherence arguments don't obviously even apply to it (though discrepancies between the creature's behavior and its makers' preferences probably produce an analogous force13 and competitive pressures probably produce a similar force for coherence in valuing resources instrumental to survival). Coherence arguments mark out an aspect of the incentive landscape, but to say that there is an incentive for something, all things equal, is not to say that it will happen.
In sum
1) Even though any behavior could be coherent in principle, if it is not coherent in combination with an entity's internal state, then coherence arguments point to a real force for different (more coherent) behavior.
2) My guess is that this force for coherent behavior is also a force for goal-directed behavior. This isn't clear, but seems likely, and also isn't undermined by Rohin's argument, as seems commonly believed.
.
Two dogs attached to the same leash are pulling in different directions. Etching by J. Fyt, 1642
.

.


AI Impacts 2020 review

Published Dec 21, 2020
This is a list of work done at AI Impacts published in 2020.
Major projects
Discontinuities project
Primary authors: Katja Grace, Richard Korzekwa, Asya Bergal, Daniel KokotajloMain page: Discontinuous progress investigationBlog post: Discontinuous progress in history: an update
Other pages added in 2020: Effect of AlexNet on historic trends in image recognition, Historic trends in transatlantic message speed, Historic trends in long-range military payload delivery, Historic trends in bridge span length, Historic trends in light intensity, Historic trends in book production, Historic trends in telecommunications performance, Historic trends in slow light technology, Penicillin and historic syphilis trends, Historic trends in the maximum superconducting temperature, Historic trends in chess AI, Effect of Eli Whitney's cotton gin on historic trends in cotton ginning, Historic trends in flight airspeed records
Evolution engineering comparison
Primary authors: Ronny FernandezMain page: How energy efficient are human-engineered flight designs relative to natural ones?
Other pages added in 2020: Energy efficiency of monarch butterfly flight, Energy efficiency of wandering albatross flight, Energy efficiency of paramotors, Energy efficiency of The Spirit of Butt's Farm, Energy efficiency of MacCready Gossamer Albatross, Energy efficiency of Boeing 747-400, Energy efficiency of Airbus A320, Energy efficiency of North American P-51 Mustang, Energy efficiency of Vickers Vimy plane, Energy efficiency of Wright model B, Energy efficiency of Wright Flyer
Time to cross human performance range
Primary authors: Richard KorzekwaPages added in 2020:
Time for AI to cross the human range in English draughtsTime for AI to cross the human range in StarCraftTime for AI to cross the human performance range in ImageNet image classificationTime for AI to cross the human performance range in chessTime for AI to cross the human performance range in Go
Historic hardware trends
Primary authors: Asya BergalPages added in 2020:
2019 recent trends in GPU price per FLOPS2019 recent trends in Geekbench score per CPU priceTrends in DRAM price per gigabyte
Interviews on plausibility of AI safety by default
Primary authors: Asya Bergal, Robert LongMain page: Interviews on plausibility of AI safety by defaultBlog post: Takeaways from safety by default interviews
Shallow survey of prescient actions
Primary authors: Richard KorzekwaMain page: Preliminary survey of prescient actions
Other work
Pages added in 2020
AI Impacts key questions of interest — Primary author: Katja GraceWas the industrial revolution a drastic departure from historic trends? — Primary author: Katja GraceSurveys on fractional progress towards HLAI — Primary author: Asya BergalPrecedents for economic n-year doubling before 4n-year doubling — Primary author: Daniel KokotajloResolutions of mathematical conjectures over time — Primary author: Asya Bergal
Blog posts added in 2020
Misalignment and misuse: whose values are manifest? — Katja GraceAutomated intelligence is not AI — Katja GraceRelevant pre-AGI possibilities — Daniel Kokotajlo, Asya BergalDescription vs simulated prediction — Richard KorzekwaAtari early — Katja GraceThree kinds of competitiveness — Daniel KokotajloAGI in a vulnerable world — Asya BergalCortes, Pizarro, and Afonso as precedents for takeover — Daniel Kokotajlo

AI Impacts 2020 review


AI Impacts key questions of interest

Updated Dec 17, 2020
This is a list of questions that AI Impacts focuses on answering.
Details
We are interested in understanding how the development of advanced AI will proceed and how it may affect humanity, especially insofar as these are relevant to efforts to improve the outcomes. This is a list of questions within this topic that we currently consider particularly important to answer.
List
AI RISK: What type and degree of risk will be posed to humanity by advanced AI systems?CHARACTER: What will early advanced AI systems be like? ARCHITECTURE: What types of algorithms will advanced AI systems use?AGENCY: Will the bulk of advanced AI systems be in the form of 'agents'?  (If so, in what sense? Will they pursue 'goals'? Can we say anything about the nature the goals or the pursuit?)PRICE: How much will the first human-level AI systems cost?TIMELINES: When will human-level AI be developed? (When will other important AI milestones take place?)TAKE-OFF SPEED: How rapid is the development of AI likely to be near human-level?DISCONTINUITY: Will there be abrupt progress in AI development at around human-level performance? INTELLIGENCE EXPLOSION: How much will AI development be accelerated by feedback from AI-based automation of the process?PRE-AI DEVELOPMENTS: What developments will take place before advanced AI is developed?PATHS TO HLAI: By what methods is advanced AI likely to come about? (e.g. will human-level AI be developed via brain emulation before it is developed via machine learning? Will neuroscientific understanding play a large role in development?)CONTEMPORARY EVENTS: How will the world be relevantly different at the time that advanced AI is developed?WARNING SIGNS: Should we expect advance notice of disruptive change from AI? (What would it look like?)POST-AI SOCIETY: If advanced AI transforms the world, what will the world look like afterwards? (e.g. What will the economic impacts be? Will humans flourish? What roles will AI systems play in society?)ACTIONS: What can we say about the impact of contemporary choices on long-term outcomes?

AI Impacts key questions of interest


How energy efficient are human-engineered flight designs relative to natural ones?

Updated Dec 10, 2020
Among two animals and nine machines:
In terms of mass⋅distance/energy, the most efficient animal was 2-8x more efficient than the most efficient machine.  All entries fell within two orders of magnitude.In terms of distance/energy, the most efficient animal was 3,000-20,000x more efficient than the most efficient machine. Both animals were more efficient than all machines. Entries ranged over more than eight orders of magnitude.
Details
Background
This case study is part of research that intends to compare the performance of human engineers and natural evolution on problems where both have developed solutions. The goal of this is to inform our expectations about the performance of future artificial intelligence relative to biological minds.
Metrics
We consider two metrics:
Distance per energy used (meters / kilojoule). Mass times distance per energy used (kilograms⋅meters / joule).
These operationalize the problem of flight into two more specific problems. There are many other aspects of flight performance that one could measure, such as energy efficiency of acceleration in a straight line, turning, hovering, vertical acceleration, vertical distance, landing, taking off, time flying per energy, and our same measures with fewer or further restrictions on acceptable entries. For instance, we might look at the problem of flying with flapping wings, or without the restriction that the solutions we consider are heavier than air and self powered.
We did not require that the flight of an entry be constantly powered. Solutions that spend some time gliding as well as some time using powered flight were allowed. Both albatrosses and butterflies use air currents to fly further.1 The energy gains from these techniques were not included in the final score, and entries were not penalized for spending a larger fraction of time gliding. It seems likely that paramotor pilots use similar techniques, since paramotors are well suited to gliding (being paragliders with propeller motors strapped to the backs of their pilots). Our energy efficiency estimate for the paramotor came from a record breaking distance flight in which the quantity of available fuel was limited, and so it is likely that some gliding was used to increase the distance traveled as much as possible.
When multiple input values could have been used, such as the takeoff weight and the landing weight, or different estimates for the energetic costs of different kinds of flight for the Monarch butterfly, we generally calculated a high and a low estimate, taking the most optimistic and pessimistic inputs respectively. In all cases, the resulting best and worst estimates differed by less than a factor of ten.
Selection of case studies
We selected case studies informally, according to judgments about possible high energy efficiencies, and with an eye to exploring a wider range of case studies.
We started by looking at the Boeing 747-400 plane, the Wandering Albatross, and the Monarch Butterfly. We chose the animals for both being known for their abilities to fly long distances, and for both having fairly different body plans.
All three scored surprisingly similarly on distance times weight per energy (details below). This prompted us to look for engineered solutions that were optimized for fuel efficiency. To that end, we looked at paramotors and record breaking flying machines. In the latter category, we found the MacCready Gassomer Albatross, which was a human powered flying device that crossed the English Channel, and the Spirit of Butts' Farm, which was a model airplane that crossed the Atlantic on one gallon of gasoline.
For reasons that are now obscure, we also included a number of different planes.
We would have liked to include microdrones, since they are different enough from other entries that they might be unusually efficient. However we did not find data on them.
Case studies
These are the full articles calculating the efficiencies of different flying machines and animals:
Wright FlyerWright model BVickers VimyNorth American P-51 MustangParamotorsThe Spirit of Butt's FarmMonarch butterflyMacCready Gossamer AlbatrossAirbus A-320Boeing 747-400Wandering albatross
Summary results
Results are available in Table 1 below, and in this spreadsheet. Figures 1 and 2 below illustrate the equivalent questions of how far each of these animals and machines can fly, given either the same amount of fuel energy, or fuel energy proportional to their body mass.

Namenatural or human-engineered kg⋅m/J    m/kJ

worstmeanbestworstmeanbest

Monarch Butterflynatural0.0650.210.36100000350000600000

Wandering Albatrossnatural1.42.<PHONE>

The Spirit of Butt's Farmhuman-engineered0.0860.120.<PHONE>

MacGready Gossamer Albatrosshuman-engineered0.<PHONE>.34.6

Paramotorhuman-engineered0.<PHONE>.10.360.360.36

Wright model Bhuman-engineered0.<PHONE>.120.10.160.21

Wright Flyerhuman-engineered0.<PHONE>.<PHONE>.130.18

North American P-51 Mustanghuman-engineered0.250.380.50.<PHONE>.092

Vickers Vimyhuman-engineered0.0810.<PHONE>.0380.05

Airbus A320human-engineered0.330.470.610.00780.00780.0078

Boeing 747-400human-engineered0.390.610.830.00210.00210.0021

Table 1: Energy efficiency of flight for a variety of natural and man-made flying entities.
Figure 1: If you give each animal or machine energy proportional to its weight, how far can it fly?
On mass⋅distance/energy, evolution beats engineers, but they are relatively evenly matched: the albatross (1.4-3.0 kg.m/J) and the Boeing 747-400 (0.39-0.83 kg.m/J) are the best in the natural and engineered classes respectively. Thus the best natural solution we found was roughly 2x-8x more efficient than the human-engineered one.2 We found several flying machines more efficient on this metric than the monarch butterfly.
Figure 2: How far animals and machines can fly on the same amount of energy. Note that the vertical axis is log scaled, unlike that of Figure 1, so smaller looking differences are in fact much larger: over eight orders of magnitude (vs less than two in Figure 1).
On distance/energy, the natural solutions have a much larger advantage. Both are better than all man-made solutions we considered. The best natural and engineered solutions respectively are the monarch butterfly (100,000-600,000 m/kJ) and the Spirit of Butts' Farm (32 m/kJ), for roughly a 3,000x to 20,000x advantage to natural evolution.

Interpretation
We take this as weak evidence about the best possible distance/energy and distance.mass/energy measures achievable by human engineers or natural evolution. One reason for this is that this is a small set of examples. Another is that none of these animals or machines were optimized purely for either of these flight metrics—they all had other constraints or more complex goals. For instance, the paramotor was competing for a record in which a paramotor had to be used, specifically. For the longest human flight, the flying machine had to be capable of carrying a human. The albatross' body has many functions. Thus it seems plausible that either engineers or natural evolution could reach solutions far better on our metrics than those recorded here if they were directly aiming for those metrics.
The measurements for distance.mass/energy covered a much narrower band than those for distance/energy: a factor of under two orders of magnitude versus around eight. Comparing best scores between evolution and engineering, the gap is also much smaller, as noted above (a factor of less than one order of magnitude versus three orders of magnitude). This seems like some evidence that that band of performance is natural for some reason, and so that more pointed efforts to do better on these metrics would not readily lead to much higher performance.
Primary author: Ronny Fernandez
Notes


Energy efficiency of monarch butterfly flight

Updated Nov 25, 2020
According to very rough estimates, the monarch butterfly:
can fly around 100,000-600,000 m/kJand move mass at around  0.065-0.36 kg⋅m/J
Details
The Monarch Butterfly is a butterfly known for its migration across North America.1

Mass
The average mass of a monarch butterfly prior to its annual migration has been estimated to be 600mg2
Distance per Joule
The following table gives some very rough estimates of energy expenditures, speeds and distances for several modes of flight, based on confusing information from a small number of papers (see footnotes for details).
ActivityDescriptionEnergy expenditure per mass ( J/g⋅hr)Energy expenditure for 600mg butterfly (J/s)Speed (m/s)distance/energy (m/J)Soaring/glidingUnpowered flight, including gradual decline and ascent via air currents8-333~0.0014 – 0.00564Very roughly 2.5-3.6 on average5446- 25716CruisingLow speed powered flightVery roughly 20970.0358Maximum: >59Maximum: >14310Sustained flappingHigh speed powered flightVery roughly 83711~0.1412 Maximum: >13.913Maximum: >9914Table 1: Statistics for several modes of flight. All figures are very rough estimates, based on incomplete and confusing information from a small number of papers (see footnotes for details).
Soaring is estimated to be potentially very energy efficient (see Table 1), since it mostly makes use of air currents for energy. It seems likely that at least a small amount of powered flight is needed for getting into the air, however monarch butterflies can apparently fly for hundreds of kilometers in a day15, so supposing that they don't stop many times in a day, taking off seems likely a negligible part of the flight.16
This would require ideal wind conditions, and our impression is that in practice, butterflies do not often fly very long distances without using at least a small amount of powered flight.17
There is stronger evidence that monarch butterflies can realistically soar around 85% of the time, from Gibo & Pallett, who report their observations of butterflies under relatively good conditions.18 So as a high estimate, we use this fraction of the time for soaring, and suppose that the remaining time is the relatively energy-efficient cruising, and take the optimistic end of all ranges. This gives us:
One second of flight = 0.15 seconds cruising + 0.85 seconds soaring
________________= 0.15s * 5 m/s cruising + 0.85s * 3.6m/s soaring
________________= 0.75m cruising + 3.06m soaring
________________= 3.81m total
This also gives us:
= 0.75m / 143 m/J cruising + 3.06m / 2571 m/J soaring
= 0.0064 J total
Thus we have:
distance/energy = 3.81m/0.0064 J = 595 m/J
For a low estimate of efficiency, we will assume that all of the powered flight is the most energetic flight, that powered flight is required half the time on average, and that the energy cost of gliding is twice that of resting. This gives us:
Energy efficiency = (50% * soaring distance + 50% * powered distance) / (50% * soaring energy + 50% * powered energy)
= (50% * soaring distance/time + 50% * powered distance/time) / (50% * soaring energy/time + 50% * powered energy/time)
= (0.5 * 2.5m/s + 0.5 * 13.9m/s) / (0.5 * (0.0056 * 2) J/s + 0.5 * 0.14 J/s)
= 108 m/J
Thus we have, very roughly:
distance/energy = 100,000-600,000 m/kJ
For concreteness, a kJ is the energy in around a quarter of a raspberry. 19
Mass⋅distance per Joule
As noted earlier, the average mass of a monarch butterfly prior to its annual migration has been estimated to be 600mg20
Thus we have:
mass*distance/energy = 0.0006 kg * 108 — 0.0006 kg * 595 m/J
= 0.065 — 0.36 kg⋅m/J

Primary author: Ronny Fernandez
Notes


Energy efficiency of wandering albatross flight

Updated Nov 25, 2020
The wandering albatross:
can fly around 240m/kJand move mass at around  1.4—3.0kg.m/J
Details
The wandering albatross is a very large seabird that flies long distances on wings with the largest span of any bird.1
Distance per Joule
Speed
In a study of wandering albatrosses flying in various wind speeds and directions, average ground speed was 12 m/s, though the fastest ground speed measured appears to be around 24m/s, 2 We use average ground speed for this estimate because we only have data on average energy expenditure, though it is likely that higher ground speeds involve more energy efficient flight, since albatross flight speed is dependent on wind and it appears that higher speeds are substantially due to favorable winds.3
Energy expenditure
One study produced an estimate that when flying, albatrosses use 2.35 times their basal metabolic rate4 which same paper implies is around 1,833 kJ/bird.day.5
That gives us a flight cost for flying of 0.050 kJ/second.6
Distance per Joule calculation
This gives us a distance per energy score of:
distance/energy
= 12 m/s / 0.050 kJ/s
= 240m/kJ
Mass.distance per Joule
Albatrosses weigh 5.9 to 12.7 kg.7
Thus we can estimate:
mass.distance/Joule
= 5.9kg * 240 m/kJ to 12.7kg*240 m/kJ
= 1.4—3.0kg.m/J
Primary author: Ronny Fernandez
Notes


Energy efficiency of paramotors

Updated Nov 24, 2020
We estimate that a record-breaking two-person paramotor:
covered around = 0.36 m/kJand moved mass at around  0.058 – 0.10 kg⋅m/J
Details
Paramotors are powered parachutes that allow the operator to steer.1
Distance per Joule
The Fédération Aéronautique Internationale (FAI) maintains records for a number of classes of paramotor contest. We look at subclass RPF2T—(Paramotors : Paraglider Control / Foot-launched / Flown with two persons / Thermal Engine)—which is appears to be the most recent paramotor record for 'Distance in a straight line with limited fuel'.2
The record distance was 123.18 km.3 The FAI rules state that no more than 7.5 kg of fuel may be used.4 We will assume that in the process of breaking this record, all of the available fuel was used. We will also assume that regular gasoline was used. Gasoline has an energy density of 45 MJ/kg.5
Distance per energy = 123.18 km / (7.5 kg * 45 MJ/kg)
= 0.36 m/kJ
Mass.distance per Joule
The weight of an entire paramotoring apparatus appears to be the weights of the passengers plus motor plus wing plus clothing and incidentals, based on forum posts.6 These posts put clothing and incidentals at around 8kg, but are estimates for single person flying, whereas this record was a two person flight. We guess that two people need around 1.5x as much additional weight, for 12kg.
Wikipedia says that the weight of a paramotor varies from 18kg to 34 kg.7 However it is unclear whether this means the motor itself, or all of the equipment involved.
The glider used appears to be MagMax brand, a typical example of which weighs around 8kg, though this may have been different in 2013, or they may have used a different specific glider.8 To account for this uncertainty, we shall add the glider weight to the high estimate, and so estimate the weight of the glider and motor together at 18-42kg.
We will assume that the apparently male pilots weighed between 65 and 115 kgs each, based on normal male weights9.
Thus we have:
weight = motor + wing + people + clothing and incidentals
weight (low estimate) = 18 + 65*2 + 12 = 160kg
weight (high estimate) = 42 + 115*2 + 12 = 284kg
High efficiency estimate:
284kg * 0.36 m/kJ  = 0.10 kg⋅m/J
Low efficiency estimate:
160kg * 0.36 m/kJ  = .058 kg⋅m/J
This gives us a range of 0.058 – 0.10 kg⋅m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of paramotors


Misalignment and misuse: whose values are manifest?

By Katja Grace, Nov 18 2020, Crossposted from world spirit sock puppet.
AI related disasters are often categorized as involving misaligned AI, or misuse, or accident. Where:
misuse means the bad outcomes were wanted by the people involved,misalignment means the bad outcomes were wanted by AI (and not by its human creators), andaccident means that the bad outcomes were not wanted by those in power but happened anyway due to error.
In thinking about specific scenarios, these concepts seem less helpful.
I think a likely scenario leading to bad outcomes is that AI can be made which gives a set of people things they want, at the expense of future or distant resources that the relevant people do not care about or do not own.
For example, consider autonomous business strategizing AI systems that are profitable additions to many companies, but in the long run accrue resources and influence and really just want certain businesses to nominally succeed, resulting in a worthless future. Suppose Bob is considering whether to get a business strategizing AI for his business. It will make the difference between his business thriving and struggling, which will change his life. He suspects that within several hundred years, if this sort of thing continues, the AI systems will control everything. Bob probably doesn't hesitate, in the way that businesses don't hesitate to use gas vehicles even if the people involved genuinely think that climate change will be a massive catastrophe in hundreds of years.
When the business strategizing AI systems finally plough all of the resources in the universe into a host of thriving 21st Century businesses, was this misuse or misalignment or accident? The strange new values that were satisfied were those of the AI systems, but the entire outcome only happened because people like Bob chose it knowingly (let's say). Bob liked it more than the long glorious human future where his business was less good. That sounds like misuse. Yet also in a system of many people, letting this decision fall to Bob may well have been an accident on the part of others, such as the technology's makers or legislators.
Outcomes are the result of the interplay of choices, driven by different values. Thus it isn't necessarily sensical to think of them as flowing from one entity's values or another's. Here, AI technology created a better option for both Bob and some newly-minted misaligned AI values that it also created—'Bob has a great business, AI gets the future'—and that option was worse for the rest of the world. They chose it together, and the choice needed both Bob to be a misuser and the AI to be misaligned. But this isn't a weird corner case, this is a natural way for the future to be destroyed in an economy.
Thanks to Joe Carlsmith for conversation leading to this post.


Energy efficiency of The Spirit of Butt's Farm

Updated Nov 18, 2020
The Spirit of Butt's Farm:
covered around 31.67 m/kJand moved mass at around 0.16 – 0.086 kg⋅m/J
Details
The Spirit of Butt's Farm was a record setting model airplane that crossed the Atlantic on one gallon of fuel.1 Fully fueled it weighed 4.987 kg, dry it weighed 2.705 kg.2
 The record setting flight used 117.1 fluid ounces of fuel.3 The straight line distance of the flight was 3,028.1 km.4 It was powered by 88% Coleman lantern fuel, mixed with lubricant.5 Coleman fuel is based on naphtha 6, so we can use the energy density of naphtha—31.4 MJ/L7—as a rough guide to its energy content, though naphtha appears to vary in its content, and it is unclear whether Coleman fuel consists entirely of naphtha.
From all this, we have:
Distance per energy = 3,028.1 km / (117.1 fl oz * 0.88 * 31.4 MJ/L)
= 31.67 m/kJ
For weight times distance per energy we will calculate a best and a worst score. To calculate the best score we will use the fully fueled weight, and to calculate the worst score we will use the dry weight. All other values are the same in both calculations.
Best score: Distance*mass/energy = 4.987 kg * 31.67 m/kJ
= 0.16 kg⋅m/J
(4.987 kg * 3,028.1km) / (117.1 US fluid ounces * 31.4MJ/litre) = 0.1389 kg*m/j
Worst score:
Distance*mass/energy = 2.705 kg * 31.67 m/kJ
= .086 kg⋅m/J
Primary author: Ronny Fernandez
Notes
Photo by Ronan Coyne, licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license, unaltered.

Energy efficiency of The Spirit of Butt’s Farm


Was the industrial revolution a drastic departure from historic trends?

Updated Nov 18, 2020
We do not have a considered view on this topic.
Details
We have not investigated this topic. This is an incomplete list of evidence that we know of:
David Roodman's analysis of the surprisingness of the industrial revolution under his 2020 model of economic history.1Ben Garfinkel's analysis of whether economic history suggests a singularity2Robin Hanson's analysis of historic economic growth understood as a sequence of exponential modes.3On a log(GWP)-log(doubling time) graph, the industrial revolution appears to be almost perfectly on trend, according to our very crude analysis.
Relevance
The nature of the industrial revolution is relevant to AI forecasting in the following ways:
If growth during the industrial revolution is a highly improbable aberration from longer term trends, it suggests that it is a consequence of specific developments at the time, most saliently new technologies. This suggests that new technologies can sometimes alone cause changes at the level of the global economy.'The impact of the industrial revolution' is sometimes used as a measure against which to compare consequences of AI developments. Thinking here may be sharpened by clarification on the nature of the industrial revolution. This use is likely related to the point above, where 'the scale of the industrial revolution' is taken to be a historically plausible scale of impact for the most ambitious technologies.If economic history is best understood as a sequence of 'growth modes',  per Hanson 2000,4 the industrial revolution being one, this changes our best extrapolation to the future. For instance, we might expect the continuation of the current mode to be slower than in a continuously super-exponential model, but may also expect to meet a new growth mode at some point, which may be substantially faster (and have other characteristics recognizable from past 'growth mode' changes). See Hanson 2000 for more on this.
Notes


Energy efficiency of MacCready Gossamer Albatross

Updated Nov 9, 2020
The MacCready Gossamer Albatross:
covered around 2.0—4.6 m/kJand moved mass at around 0.1882 —0.4577 kg⋅m/J
Details
The MacCready Gossamer Albatross was a human-powered flying machine that crossed the English Channel in 1979.1 The pilot pedaled the craft, seemingly as if on a bicycle. It had a gross mass of 100kg, flying across the channel,2 and flew 35.7 km in  2 hours and 49 minutes.3 The crossing was difficult however, so it seems plausible that the Gossamer Albatross could fly more efficiently in better conditions.
We do not know the pilot's average power output, however:
Wikipedia claims at least 300W was required to fly the craft4Chung 2006, an engineering textbook, claims that the driver, a cyclist, could produce around 200W of power.5Our impression is that 200W is a common power output over houres for amateur cycling. For instance, one of our researchers is able to achieve this for three hours.6
The best documented human cycling wattage that we could easily find is from professional rider Giulio Ciccone who won a stage of the Tour de France, then uploaded power data to the fitness tracking site Strava.7 His performance suggests around 318W is a reasonable upper bound, supposing that the pilot of the Gossamer Albatross would have had lower performance.8
To find the energy used by the cyclist, we divided power output by typical efficiency for a human on a bicycle, which according to Wikipedia ranges from .18 to .26.9
Distance per Joule
For distance per energy this gives us a highest measure of:
35.7 km / ((200W * (2 hours + 49 minutes))/0.26) = 4,577 m/MJ
And a lowest measure of:
35.7 km / ((318W * (2 hours + 49 minutes))/0.18) = 1,993 m/MJ
Mass per Joule
For weight times distance per energy this gives us a highest measure of:
(100kg * 35.7 km) / ((200W * (2 hours + 49 minutes))/0.26) = 0.4577 kg⋅m/j
And a lowest measure of:
(100kg * 35.7 km) / ((318W * (2 hours + 49 minutes))/0.17) =  0.1882 kg⋅m/j
Primary author: Ronny Fernandez
Notes


Energy efficiency of Boeing 747-400

Updated Nov 5, 2020
The Boeing 747-400:
covers around 0.0021m/kJ.and moves mass at around  0.39 – 0.83 kg.m/J
Details
The Boeing 747-400 is a 1987 passenger plane.1
Distance per Joule
The plane uses 10.77 kg/km of fuel on a medium haul flight.2 We do not know what type of fuel it uses, but typical values for aviation fuel are around 44MJ/kg.3 Thus to fly a kilometer, the plane needs 10.77 kg of fuel, which is 10.77 x 44 MJ = 474 MJ of fuel. This gives us 0.0021m/kJ.
Mass.distance per Joule
According to Wikipedia, the 747's 'operating empty weight' is 183,523 kg and its 'maximum take-off weight' is 396,893 kg.4 We use the range 183,523 kg—396,893 kg since we do not know at what weight in that range the relevant speeds were measured.
We have:
Distance per kilojoule: 0.0021m/kJMass: 183,523 kg—396,893 kg
This gives us a range of 0.39 – 0.83 kg.m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of Boeing 747-400


Energy efficiency of Airbus A320

Updated Nov 5, 2020
The Airbus A320:
covers around 0.0078 m/kJand moves mass at around  0.33 – 0.61 kg.m/J
Details
The Airbus A320 is a 1987 passenger plane.1
Distance per Joule
The plane uses 2.91 kg of fuel per km on a medium haul flight.2 We do not know what type of fuel it uses, but typical values for aviation fuel are around 44MJ/kg.3 Thus to fly a kilometer, the plane needs 2.91kg of fuel, which is 2.91 x 44 MJ = 128MJ of fuel. This gives us 0.0078 m/kJ
Mass.distance per Joule
According to modernairliners.com, the A320's 'operating empty weight' is 42,600 kg and its 'maximum take-off weight' is 78,000 kg.4 We use the range 42,600—78,000 kg, since we do not know at what weight in that range the relevant speeds were measured.
We have:
Distance per kilojoule: 0.0078 m/kJMass: 42,600—78,000 kg
This gives us a range of 0.33 – 0.61 kg.m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of Airbus A320


Energy efficiency of North American P-51 Mustang

Updated Nov 5, 2020
The North American P-51 Mustang:
flew around 0.073—0.092 m/kJand moved mass at around  0.25 – 0.50 kg.m/J
Details
The North American P-51 Mustang was a 1940 US WWII fighter and fighter-bomber.1
Mass
According to Wikipedia2:
Empty weight: 7,635 lb (3,465 kg)Gross weight: 9,200 lb (4,175 kg)Max takeoff weight: 12,100 lb (5,488 kg)
We use the range 3,465—5,488 kg, since we do not know at what weight in that range the relevant speeds were measured.
Distance per Joule
Wikipedia tells us that cruising speed was 362 mph (162 m/s)3
A table from WWII Aircraft Performance gives combinations of flight parameters apparently for a version of the P-51, however it has no title or description, so we cannot be confident. 4 We extracted some data from it here. This data suggests the best combination of parameters gives a fuel economy of 6.7 miles/gallon (10.8km)
We don't know what fuel was used, but fuel energy density seems likely to be between 31—39 MJ/L  = 117—148 MJ/gallon.5
Thus the plane flew about 10.8km on 117—148 MJ of fuel, for 0.073—0.092 m/kJ
Mass.distance per Joule
We have:
Distance per kilojoule: 0.073—0.092 m/kJMass: 3,465—5,488 kg
This gives us a range of 0.25 – 0.50 kg.m/J
Primary author: Ronny Fernandez
Notes


Energy efficiency of Vickers Vimy plane

Updated Nov 5, 2020
The Vickers Vimy:
flew around 0.025—0.050 m/kJand moved mass at around  0.081 – 0.25 kg.m/J
Details
The Vickers Vimy was a 1917 British WWI bomber.1 It was used in the first non-stop transatlantic flight.
Mass
According to Wikipedia2:
Empty weight: 7,104 lb (3,222 kg)Max takeoff weight: 10,884 lb (4,937 kg)
We use the range 3,222—4,937 kg, since we do not know at what weight in that range the relevant speeds were measured.
Energy use per second
We also have:
Power: 360 horsepower = 270 kW3Efficiency of use of energy from fuel: we did not find data on this, so use an estimate of 15%-30%, based on what we know about the energy efficiency of the Wright Flyer.
From these we can calculate:
Energy use per second = power of engine x 1/efficiency in converting energy to engine power = 270kJ/s / .15—270kJ/s / .30 = 900—1800 kJ/s
Distance per second
Wikipedia gives us:
Maximum speed: 100 mph (160 km/h, 87 kn)4
Note that the figures for power do not obviously correspond to the highest measured speed. This is a rough estimate.
Distance per Joule
We now have (from above):
speed = 100 miles/h = 44.7m/senergy use = 900—1800 kJ/s
Thus, on average each second the plane flies 44.7 m and uses 900—1800 kJ, for 0.025—0.050 m/kJ.
Mass.distance per Joule
We have:
Distance per kilojoule: 0.025—0.050 m/kJMass: 3,222—4,937 kg
This gives us a range of 0.081 – 0.25 kg.m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of Vickers Vimy plane


Energy efficiency of Wright model B

Updated Nov 5, 2020
The Wright model B:
flew around 0.10—0.21m/kJand moved mass at around 0.036 – 0.12 kg.m/J
Details
The Wright Model B was a 1910 plane developed by the Wright Brothers.1
Mass
According to Wikipedia2:
Empty weight: 800 lb (363 kg)Gross weight: 1,250 lb (567 kg)
We use the range 363—567 kg, since we do not know at what weight in that range the relevant speeds were measured.
Energy use per second
From Wikipedia, we have3:
Power: 35 horsepower = 26kWEfficiency of use of energy from fuel: we could not find data on this, so use an estimate of 15%-30%, based on what we know about the energy efficiency of the Wright Flyer.
From these we can calculate:
Energy use per second = power of engine x 1/efficiency in converting energy to engine power = 26kJ/s / .15—26kJ/s / .30 = 86.6—173 kJ/s
Distance per second
Wikipedia gives us4:
Maximum speed: 45 mph (72 km/h, 39 kn)Cruise speed: 40 mph (64 km/h, 35 kn)
We use the cruise speed, as it seems more likely to represent speed achieved with the energy usages reported.
Distance per Joule
We now have (from above):
speed = 40miles/h = 17.9m/senergy use = 86.6—173 kJ/s
Thus, on average each second the plane flies 17.9m and uses 86.6—173 kJ, for 0.10—0.21m/kJ.
Mass.distance per Joule
We have:
Distance per kilojoule: 0.10—0.21m/kJMass: 363—567kg
This gives us a range of 0.036 – 0.12 kg.m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of Wright model B


Energy efficiency of Wright Flyer

Updated Dec 10, 2020
The Wright Flyer:
flew around 0.080-0.18m/kJand moved mass at around .022 – .061 kg.m/J
Details
The Wright Flyer (Flyer I) was the first successful plane, built in 1903.1
Mass
According to Wikipedia2:
Empty weight: 605 lb (274 kg)Max takeoff weight: 745 lb (338 kg)
Energy use per second
Fuel use per hour
A 1904 article in the Minneapolis Journal says the plane consumed 'a little less than  than ten pounds of gasoline per hour'.3 A pound of gasoline contains around 20MJ4 So we have:
Hourly fuel consumption: 10lb/h x 20MJ/lb = 200MJ/h = 55kJ/s
We don't know how reliable this source is. For instance, a 1971 book, The Write Brothers' Engines and their Design does not give data on fuel consumption in their table of engine characteristics for lack of available comprehensive data5, suggesting that its authors did not consider the article strong evidence, though it is also possible that they didn't have access to the 1904 article.
Utilized motor power / efficiency
To confirm, we can estimate the plane's energy use per second by a second means: combining the claimed power (energy/second) made use of by the engine, and a guess about how much fuel is needed to deliver that amount of energy.
Utilized motor power
According to Wikipedia, the plane had a 12 horsepower (9 kJ/s), gasoline engine.6 The 1904 Minneapolis Journal article put it at 16 horsepower (12 kJ/s), and Orville Wright, quoted by Hobbs (1971), puts it at 'almost 16 horsepower' at one point.7 Hobbs says that at one point this engine achieved 25 horsepower, though this probably isn't representative of what was 'actually utilized'.8 For that he gives a range of 8.25-16 horsepower.9 In light of these estimates, we will use 8.25-16 horsepower, which is 6.15-12 kJ/s.
Efficiency of energy conversion from fuel to motor power
A quote from Orville Wright suggests fuel consumption as 0.580lb of fuel per horsepower hour.10 This would imply 23% of energy was utilized from the fuel.11
Hobbs notes that this seems low, but assumes a similar efficiency: 24.50%. Thus he presumably doesn't find it implausible.12
According to Wikipedia, the thermal efficiency of a typical gasoline engine is 20%13. It seems that this has increased14, which would suggest that the typical figure was lower in 1903. We don't think this undermines the more specific figures given above.
It seems likely that Hobbs number is best here, since he knows about Wright's number, and may have more information than us about for instance the exact fuel being used. So we use 24.5%.
Calculation of energy use from motor power/efficiency
Combining these numbers, we have:
power spent = 6.15-12 kJ/s used by engine x 1 / 24.5% fuel energy needed to get one unit of energy used by engine, given inefficiency
= 25-49 kJ/s
Calculation of energy use
We now have:
energy expenditure calculated via motor power and efficiency: 25-49 kJ/senergy expenditure calculated via hourly fuel use: 55kJ/s
Neither figure seems clearly more reliable, so we will use the range 25-55kJ/s
Distance per second
Two of the Wright Flyer's first flights were 120 feet in 12 seconds and 852 feet in 59 seconds.15 This gives speeds of 3m/s and 4.4m/s. We will use the second, since it better represents successful flight, still within the first days.
Distance per Joule
We now have (from above):
speed = 4.4 m/senergy use = 25-55 kJ/s
Thus, on average each second the plane flies 4.4m and uses 25-55kJ, for 0.080-0.18m/kJ.
Mass.distance per Joule
We have:
Distance per kilojoule: 0.080-0.18m/kJMass: 274-338kg
This gives us a range of .022 – .061 kg.m/J
Primary author: Ronny Fernandez
Notes

Energy efficiency of Wright Flyer


Automated intelligence is not AI

By Katja Grace, Nov 1 2020, Crossposted from world spirit sock puppet.
Sometimes we think of 'artificial intelligence' as whatever technology ultimately automates human cognitive labor.
I question this equivalence, looking at past automation. In practice human cognitive labor is replaced by things that don't seem at all cognitive, or like what we otherwise mean by AI.
Some examples:
Early in the existence of bread, it might have been toasted by someone holding it close to a fire and repeatedly observing it and recognizing its level of doneness and adjusting. Now we have machines that hold the bread exactly the right distance away from a predictable heat source for a perfect amount of time. You could say that the shape of the object embodies a lot of intelligence, or that intelligence went into creating this ideal but non-intelligent tool.Self-cleaning ovens replace humans cleaning ovens. Humans clean ovens with a lot of thought—looking at and identifying different materials and forming and following plans to remove some of them. Ovens clean themselves by getting very hot.Carving a rabbit out of chocolate takes knowledge of a rabbit's details, along with knowledge of how to move your hands to translate such details into chocolate with a knife. A rabbit mold automates this work, and while this route may still involve intelligence in the melting and pouring of the chocolate, all rabbit knowledge is now implicit in the shape of the tool, though I think nobody would call a rabbit-shaped tin 'artificial intelligence'.Human pouring of orange juice into glasses involves various mental skills. For instance, classifying orange juice and glasses and judging how they relate to one another in space, and moving them while keeping an eye on this. Automatic orange juice pouring involves for instance a button that can only be pressed with a glass when the glass is in a narrow range of locations, which opens an orange juice faucet running into a spot common to all the possible glass-locations.
Some of this is that humans use intelligence where they can use some other resource, because it is cheap on the margin where the other resource is expensive. For instance, to get toast, you could just leave a lot of bread at different distances then eat the one that is good. That is bread-expensive and human-intelligence-cheap (once you come up with the plan at least). But humans had lots of intelligence and not much bread. And if later we automate a task like this, before we have computers that can act very similarly to brains, then the alternate procedure will tend to be one that replaces human thought with something that actually is cheap at the time, such as metal.
I think a lot of this is that to deal with a given problem you can either use flexible intelligence in the moment, or you can have an inflexible system that happens to be just what you need. Often you will start out using the flexible intelligence, because being flexible it is useful for lots of things, so you have some sitting around for everything, whereas you don't have an inflexible system that happens to be just what you need. But if a problem seems to be happening a lot, it can become worth investing the up-front cost of getting the ideal tool, to free up your flexible intelligence again.

Automated intelligence is not AI


Time for AI to cross the human range in English draughts

Updated 26 Oct 2020
AI progress in English draughts performance crossed the following ranges in the following times:
RangeStartEndDuration (years)First attempt to beginner level1951~1956, <1961~4, <10Beginner to superhuman~1956, <<PHONE>~38, >33Above superhuman19942007*13** treating perfect play as the end of progress, though progress could potentially be made in performing better against imperfect play.

Details
Metric
'English Draughts' is a popular variety of draughts, or checkers.
Here we look at direct success of AI in beating human players, rather than measuring humans and AI on a separate metric of strength.
Data
Data here comes mostly from Wikipedia. We found several discrepancies in Wikipedia's accounts of these events, so consider the remaining data to be somewhat unreliable.
AI achievement of human milestones
Earliest attempt
According to Wikipedia, the first checkers program was run in 1951.1
Beginner level
There seems to be some ambiguity around the timing and performance of Arthur Samuel's early draughts programs, but it appears that he worked on them from around 1952. In 1956, he demonstrated a program on television. 2 It is unclear how good the program's play was, but it is said to have resulted in a fifteen-point rise in the stock price of IBM when demonstrated to IBM shareholders, seemingly at around that time. This weakly suggests that the program played at at least beginner level.
In 1962 Samuel's program apparently beat an ambiguously skilled player who would by four-years later become a state champion in Connecticut.3 Thus progress was definitely beyond beginner level by 1962.
Superhuman level
In 1994, computer program Chinook drew six times against world champion Marius Tinsley, before Tinsley withdrew due to pancreatic cancer and Chinook officially won.4 Thus Chinook appears to have been close to as good as the best player in 1994.
End of progress
In 2007 checkers was 'weakly solved', which is to say that perfect play guaranteeing a draw for both sides from the start of the game is known, from the starting state5 (this does not imply that if someone plays imperfectly, perfect moves following this are known).6 This is not the best possible performance by all measures, since further progress could presumably be made on reliably beating worse players.
Times for AI to cross human-relative ranges
Given the above dates, we have:
RangeStartEndDuration (years)First attempt to beginner level1951~1956, <1961~4, <10Beginner to superhuman~1956, <<PHONE>~38, >33Above superhuman19942007*13** treating perfect play as the end of progress, though progress could potentially be made in performing better against imperfect play.

Primary author: Katja Grace
Notes


Time for AI to cross the human range in StarCraft

Published 20 Oct 2020; updated 22 Oct 2020
Progress in AI StarCraft performance took:
~0 years to reach the level of an untrained human~21 years to pass from beginner level to high professional human level~2 years to continue from trained human to current performance (2020), with no particular end in sight.
Details
Metric
We compare human and AI players on their direct ability to beat one another (rather than a measure of the overall performance of each).
AI milestones
Earliest attempt
Starcraft was released in 19981 The game allows the player to play against a computer opponent, however this built-in AI has access to information that a normal player would not have. For instance, it has real-time information about what the other player is doing at all times, which is normally hidden. We do not have detailed knowledge about early StarCraft AIs that do not have this advantage, but our impression is that it was possible to write them from the start (see next section).
Beginner level
Our impression is that since StarCraft Brood War came out in 1998, it has been possible to write a bot that can beat a player who recently learned the game, without "cheating" in the way that the game's built-in computer opponents do. This is uncertain, and based on private discussion with people who write Starcraft AIs that compete in tournaments.
Professional level
In 2018, DeepMind's AlphaStar beat MaNa2, a strong professional player (seemingly 13th place in the 2018 StarCraft II World Championship Series Circuit)3. This does not imply that AlphaStar was in general a stronger player than MaNa, but suggests AlphaStar was at a broadly high professional level. How  AlphaStar's performance compares to humans will depend on how narrowly the task is defined, so that if the AI is not allowed to give commands faster than a human is able, it will compare less favorably than if it is allowed to give commands very quickly4.
Times for AI to cross human-relative ranges
Given the above dates, we have:
RangeStartEndDuration (years)First attempt to beginner level19981998~0Beginner to superhuman19982018~21Above superhuman2018?>2

Primary author: Rick Korzekwa
Notes


Time for AI to cross the human performance range in ImageNet image classification

Published 19 Oct 2020
Progress in computer image classification performance took:
Over 14 years to reach the level of an untrained human3 years to pass from untrained human level to trained human level5 years to continue from trained human to current performance (2020)
Details
Metric
ImageNet1 is a large collection of images organized into a hierarchy of noun categories. We looked at 'top-5 accuracy' in categorizing images. In this task, the player is given an image, and can guess five different categories that the image might represent. It is judged as correct if the image is in fact in any of those five categories.
Human performance milestones
Beginner level
We used Andrej Karpathy's interface2 for doing the ImageNet top-5 accuracy task ourselves, and asked a few friends to do it. Five people did it, with performances ranging from 74% to 89%, with a median performance of 81%.
This was not a random sample of people, and conditions for taking the test differed. Most notably, there was no time limit, so time allocated was set by patience for trying to marginally improve guesses.
Trained human-level
ImageNet categorization is not a popular activity for humans, so we do not know what highly talented and trained human performance would look like. The best relatively high human performance measure we have comes from Russakovsky et al, who report on performance of two 'expert annotators', who they say learned many of the categories. 3 The better performing annotator there had a 5.1% error rate.4
AI achievement of human milestones
Earliest attempt
The ImageNet database was released in 2009.5. An annual contest, the ImageNet Large Scale Visual Recognition Challenge, began in 2010.6
In the 2010 contest, the best top-5 classification performance had 28.2% error.7
However image classification broadly is older. Pascal VOC was a similar previous contest, which ran from 2005.8 We do not know when the first successful image classification systems were developed. In a blog post, Amidi & Amidi point to LeNet as pioneering work in image classification9, and it appears to have been developed in 1998.10
Beginner level
The first entrant in the ImageNet contest to perform better than our beginner level benchmark was SuperVision (commonly known as AlexNet) in 2012, with a 15.3% error rate.11
Superhuman level
In 2015 He et al apparently achieved a 4.5% error rate, slightly better than our high human benchmark.12
Current level
According to paperswithcode.com, performance has continued to climb, to 2020, though slower than earlier.13
Times for AI to cross human-relative ranges
Given the above dates, we have:
RangeStartEndDuration (years)First attempt to beginner level<<PHONE>>14Beginner to superhuman201220153Above superhuman2015>2020>5

Primary author: Rick Korzekwa
Notes


Time for AI to cross the human performance range in Go

Posted 15 Oct 2020; updated 19 Oct 2020
Progress in computer Go performance took:
0-19 years to go from the first attempt to playing at human beginner level (<1987)>30 years to go from human beginner level to superhuman level (<<PHONE>)3 years to go from superhuman level to the the current highest performance (2017-2020)
Details
Human performance milestones
Human go ratings range from 30 kyu (beginner), through 7 dan to at least 9 professional dan.1 These ratings go downwards through kyu levels, then upward through dan levels, then upward through professional dan levels. The top ratings seem to be closer together than the lower ones, though there are apparently multiple systems which vary)2
AI achievement of human milestones
Earliest attempt
Wikipedia says the first Go program was written in 1968.3 We do not know how well it performed.
Beginner level
We have not investigated early Go performance in depth. Figure 1 includes informed guesses about early performance by David Fotland, author of successful Go program, The Many Faces of Go, and Sensei's Library, a Go wiki.4 Fotland says that early data on AI Go performance is poor, since bots did not play in tournaments, so were not rated.
Figure 1: From Grace 2013.
This suggests that by 1987 Go bots were performing better than human beginners. We do not have evidence to pin down the date of human beginner level AI better, but have also not investigated thoroughly (there appears to be more evidence).
Superhuman level
In May 2017 AlphaGo beat the top ranked Go player in the world.5 This does not imply that AlphaGo was overall better, but a new version in October could beat the May version in 89 games out of 1006, suggesting that if in May it would have beaten Ke Jie in more than 11% of games, the new version would beat Ke Jie more than half the time, i.e. perform better than the best human player. Thus 2017 seems like a reasonable date for top human-level play.
Times for AI to cross human-relative ranges
Given the above dates, we have:
RangeStartEndDuration (years)First attempt to beginner level1968<1987<19Beginner to superhuman<<PHONE>>30Above superhuman2017>2020>3

Primary author: Katja Grace
Notes


Time for AI to cross the human performance range in chess

Published 15 Oct 2020
Progress in computer chess performance took:
~0 years to go from from playing chess at all to playing it at human beginner level~49 years to go from human beginner level to superhuman level~11 years to go from superhuman level to the the current highest performance
Details
Human range performance milestones
We use the common Elo system for measuring chess performance. Human chess Elo ratings range from around 800 (beginner)1 to 2882 (highest recorded).2 The highest recorded human score is likely higher than it would have been without chess AI existing, since top players can learn from the AI.3
Times for machines to cross ranges
Beginner to superhuman range
We could not find transparent sources for low computer chess Elo records, but it seems common to place Elo scores of <PHONE> in the 1950s and 1960s. In his book Robot (1999)[/note]Moravec, Hans. Robot: Mere Machine to Transcendent Mind, n.d., p71, also at https://frc.ri.cmu.edu/~hpm/book97/ch3/index.html[/note], Moravec gives the diagram shown in Figure 1, which puts a machine with an Elo of around 800 in 1957. He does not appear to provide a source for this however. Figure 2 shows another figure without sources from a 2002 article by L. Stephen Coles at Dr. Dobbs 4, which puts some machine at over 1000 in around 1950. To err on the side of assuming narrow human ranges, and because Moravec appears to be a more reliable source, we use his data here. This means that machine chess performance entered the human range in 1957 at the latest.
Figure 1: Graph from Moravec, 19995. We did not find Moravec's sources for these numbers.
Figure 2: Chess AI progress compared to human performance, from Coles 20026. The image appears to confusingly claim that the present time is around 1993, in which case the right of the graph (after 'now') must be imagined, though it appears to be approximately correct.
The chess computer Deep Blue famously beat the then world-champion Kasparov under tournament conditions in 1997.7 However this does not imply that Deep Blue was at that point overall more capable than Kasparov, i.e. had a higher Elo rating.8
According to the Swedish Chess Computer Association records, 2006 is the year when the highest machine Elo rating surpassed the highest human Elo (both the highest at the time, and the highest in 2020). In particular Rybka 1.2 was rated 2902. At the time, the highest human Elo rating was Garry Kasparov at 2851.9
Thus it took around 49 years for computers to progress from beginner human level chess to superhuman chess.
Pre-human range
The Chess Programming Wiki says that the 1957 Bernstein Chess program was the first complete chess program.10 This seems likely to be the same Bernstein program noted by Moravec as having an 800 Elo in 1957 (see above). Thus if correct, this means that once machines could complete the task of playing chess at all, they could already do it at human beginner level. This may not be accurate (none of these sources appear to be very reliable), but it strongly suggests that the time between lowest possible performance and beginner human performance was not as long as decades.
Superhuman performance range
The Swedish Chess Computer Association has measured continued progress. As of July 2020, the best chess machine is rated 355811, whereas in 2019 sometime, the highest rating was 3529.12 Alphazero also appeared to have an Elo just below 3500 in 2017, according to its creators (from a small figure with unclear labels).13
We know of no particular upper bound to chess performance.
This suggests that so far the superhuman range in chess playing has permitted at least least 14 years of further progress, and may permit much more.
Primary author: Katja Grace
Notes