Historic trends in telecommunications performance

Published February 2020
January 2023 note: This page contains errors that have not yet been corrected. Our overall conclusion, that there were likely no discontinuities in our metrics for telecommunications performance are likely unaffected by these errors.
There do not appear to have been any greater than 10-year discontinuities in telecommunications performance, measured as:

bandwidth-distance product for all technologies 1840-2015
bandwidth-distance product for optical fiber 1975-2000
total bandwidth across the Atlantic 1956-2018

Radio does not seem likely to have represented a discontinuity in message speed.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
Fiber optic cables were first used for telecommunications in the 1970s and 80s.1 While previous telecommunications technology sent information via electricity, fiber optic cables instead sent information via light. Though electric signals travel at around 80% of the speed of light, while optical signals within a fiber only travel at roughly 70% of the speed of light, fiber optics have other benefits which add up to a considerable advantage.2
Fiber optic cable: laser light shining on one end comes out of the other.3
Trends
Bandwidth-distance product, usually given in bits*kilometers/seconds, is both the most apparently relevant metric of progress in telecommunications4 5, and the one that was suggested to us as discontinuous. We also considered data transfer rate (measured in Mbps) for transatlantic cables, as a metric which more closely tracks the performance of cables that were actually in use, with the Atlantic serving as a distance constraint. We found separate data for bandwidth-distance product across all technologies, in fiber optics alone, and crossing the Atlantic, so we consider each of these metrics.
Bandwidth-distance product across all technologies 1840-2015
Data
We used a tool for extracting data from figures6 to extract data from Figure 8.2 from Agrawal, 2016,7 shown in Figure 1. We put the data into this spreadsheet. Figures 2 and 3 show this data without a trendline, and the log of the data on a log axis with a straight trendline.
Figure 1 below shows progress in bandwidth-distance product across all technologies on a log scale.
Figure 1: Growth in bandwidth-distance product across all telecommunications during 1840-2015 from Agrawal, 2016
Figure 2: Agrawal’s data, manually extracted, without trendline.
Figure 3: Log of Agrawal’s data, shown on a log axis. The linear fit says that the data is well modeled as double exponential.
Discontinuity measurement
If we treat the previous rate of progress at each point to be exponential (as Agrawal does, with two different regimes) then optical fibers appear to represent a 27 year discontinuity.8 The following 2-3 developments are also substantial discontinuities, depending on whether one breaks the data into multiple trends. As shown in Figure 3 however, the log of the data fits an exponential trend well. If we extrapolate progress expecting the log to be exponential, there are no discontinuities of more than ten years in this data. This seems like the better fit, so we take it there are not discontinuities.
Argawal’s data also does not include minor improvements on the broad types of systems mentioned, which presumably occurred. In particular, our impression is that there were better coaxial cables as well as worse optical fibers, such that the difference when fiber optics appeared was probably not more than a factor of two,9 10 or about six years of exponential progress at the rate seemingly prevailing around the time of coaxial cables.11
Bandwidth-distance product in fiber optics alone 1975-2000
Data
We used a tool for extracting data from figures to extract data from Figure 8.8 from Agrawal, 201612 and put it into this spreadsheet.
Figure 4 below shows bandwidth-distance product on a log scale in fiber optics alone, from Agrawal, 2016.
Figure 4:  Progress in bandwidth-distance product in fiber optics alone, from Agrawal, 2016 (Note: 1 Gb = 10^9 bits)
Discontinuity measurement
We chose to model this data as a single exponential trend.13 Compared to previous rates in this trend, there are no greater than ten year discontinuities in bandwidth-distance product in fiber optics alone.14
Bandwidth for Transatlantic Cables 1956-2018
Data
Figure 5 shows bandwidth of transatlantic cables according to our own calculations, based on data we collected mainly from Wikipedia.15
Figure 5: Transatlantic cable bandwidth of all types. Pre-1980 cables were copper, post-1980 cables were optical fiber.
Discontinuity measurement
We treat this data as a single exponential trend.16 The data did not contain any discontinuities of more than ten years.18
There was a notable temporary increase in the growth rate between 1996 and 2001. We speculate that this and the following 15 years of stagnation may be a result of heavy telecommunications investment during the dot com bubble.19
Notes


Historic trends in slow light technology

Published Feb 7 2020
Group index of light appears to have seen discontinuities of 22 years in 1995 from Coherent Population Trapping (CPT) and 37 years in 1999 from EIT (condensate). Pulse delay of light over a short distance may have had a large discontinuity in 1994 but our data is not good enough to judge. After 1994, pulse delay does not appear to have seen discontinuities of more than ten years.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
That which is uncited on this page is our understanding, given familiarity with the topic.1
“Slow Light” is a phenomenon where the speed at which a pulse of light propagates through a medium is greatly reduced. This has potential applications for lasers, communication, and cameras.2
The speed of propagation of the light through the medium referred to as the ‘group velocity‘ of the light, and it is a function of the medium’s refractive index and dispersion (the rate at which the refractive index changes with the frequency of the light).
In most materials—for instance glass, air, or water—the dispersion is low enough that the group velocity is simply the speed of light divided by the index of refraction. In order to slow down light by more than roughly a factor of 3, physicists needed to create optical media with a greater dispersion in the frequency range of interest. The challenge in this was doing so without the medium absorbing most of the light, since most materials exhibit maximum dispersion under conditions of high absorption. This was resolved using exotic phases of matter and sophisticated methods for inducing transparency in them.
Summary of historic developments
Diamonds have a very high index of refraction, and the ability to cut and polish them to achieve good optical quality has existed for hundreds of years3. However there were no light sources available for studying low group velocities until the 1960s, so recorded progress begins then. The first pulsed sources of light that could reasonably be used for the investigation of slow light came about in 1962 with the invention of Q-switching, which is a method for generating series of short light pulses from a laser. We do not know whether early Q-switched lasers could be used for this work, but doubt that any earlier light sources were suitable.
Following Q-switching, progress for slowing light proceeded roughly in four stages:
High index materials: For instance, diamonds. There may have been marginally better materials, but we did not investigate because our understanding is that they should at most represent a few tens of a percent of difference, and later gains represent factors of millions to trillions.High absorption media: Materials with very low group velocity at a particular wavelength range, at the cost of very high absorption (losing >99% of the light over <100 microns).Induced transparency: Materials with a narrow window of transparency in spectral regions of low group velocity. This led to rapid increases in total delay of a pulse, both through longer propagation distance and lower speeds.Stopped light: Eventually, group velocity had been lowered to the point that it was possible to destroy the pulse, but store enough information in the medium about it to reconstruct it after some delay. There is room for debate about whether this is really the same pulse of light, however there are applications in which treating is as such is reasonable. We view this as progress in pulse delay, but not group index. After the invention of stopped light, slow light was no longer a major target for progress.
Trends
There are several metrics that one might plausibly be interested in in this area. Group velocity is a natural choice because it is simple, but it trades off against absorption. So it is relatively easy to make a medium that has a very low group velocity, but it will absorb too much light to be useful. Because of this, research was more plausibly aimed at some combination of low  group velocity and low absorption.
One simple way to combine absorption and group velocity into a single metric is group velocity with an absorption criterion (say, lowest group velocity in a medium that transmits at least 1% of the light). Another is total time delay of the pulse by the medium, since longer delays can be achieved either by slowing down the pulse more, or slowing it down over a longer distance (requiring lower absorption). Pulse delay seems to have been a goal for researchers, suggesting it tracks something important, making it more interesting from our perspective.
We chose to investigate pulse delay and group index (the speed of light divided by the group velocity).4
Pulse delay and group index
Data
We collected data from a variety of online sources into this spreadsheet. The sheet shows progress in pulse delay and group index over time as well as our source for each data point, and calculates unexpected progress at each step. Figures 1-3 illustrates these trends.
Figure 1: Progress in delay of a pulse of light over a short distance
 Figure 2: Progress in group index of a material (speed of light divided by speed of light in that material)
Figure 3: Progress in pulse delay and group index. “Human speed” shows the rough scale of motion familiar to humans.
Discontinuity measurement
For comparing points to ‘past rates of progress’ we treat past progress for both pulse delay and group index as exponential, changing to a new exponential regime near 1995 in both cases.5
Compared to these rates of past progress, the 1994 point—EIT (hot gas)—could be a very large discontinuity in pulse delay, if there was a small amount of progress prior to it. There probably was, however our estimates of the points leading up to it are so uncertain that it isn’t clear that there was any well-defined progress, and if there was we have not measured it. So we do not attempt to judge whether there is a discontinuity there. Aside from that, pulse delay saw no discontinuities of more than ten years.
Group index has discontinuities of 22 years in 1995 from CPT and 37 years in 1999 from EIT (condensate).
In addition to the size of these discontinuities in years, we have tabulated a number of other potentially relevant metrics here.6
Discussion of causes
These trends are short and not characterized by a clearly established rate prior to any potential change of rate, making changes in apparent rate relatively unsurprising. This means they are both less in need of explanation, and less informative about what to expect in cases where a technology does have a better-established progress trend.
Increasing group index of light does not appear to have been a major research goal prior to the discovery of induced transparency in the mid 1990s. Most of the work up to that point (and, to a lesser, extent after) was directed toward controlling the properties of optical media in general, with group index as one particularly salient parameter that could be controlled, but perhaps at the expense of others. Thus the moderate discontinuities in group index might relate to the hypothesized pattern of metrics that receive ongoing concerted effort tending to be more continuous than those receiving weak or sporadic attention.
Primary author: Rick Korzekwa
Thanks to Stephen Jordan for suggesting slow light as a potential area of discontinuity.
Notes


Penicillin and historic syphilis trends

Penicillin did not precipitate a discontinuity of more than ten years in deaths from syphilis in the US. Nor were there other discontinuities in that trend between 1916 and 2015.
The number of syphilis cases in the US also saw steep decline but no substantial discontinuity between 1941 and 2008.
On brief investigation, the effectiveness of syphilis treatment and inclusive costs of syphilis treatment do not appear to have seen large discontinuities with penicillin, but we have not investigated either thoroughly enough to be confident.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
Penicillin was first used to treat a patient in 19411 and became mass-produced in the US between 1942 and 1944.2 It quickly became the preferred treatment for syphilis, and appears to be generally credited with producing a steep decline in the prevalence of syphilis which was seen at around that time.3 4
Figure 1: US World War II Poster5
Trends
We consider four metrics of success in treating syphilis: the number of syphilis cases, the number of syphilis deaths, effectiveness of syphilis treatment, and the inclusive cost of treatment.
In addition to the size of any discontinuities in years, we tabulated a number of other potentially relevant statistics for each metric here.
US Syphilis cases
Data
Figure 1 shows historic reported syphilis cases after 1941, according to the CDC.6 We converted the data in the figure into this spreadsheet.7

Figure 1: Syphilis—Reported Cases by Stage of Infection, United States, 1941–2009, according to the CDC8
Discontinuity Measurement
According to this data, total cases of syphilis declined by around 80% over fifteen years (see Figure 1). We do not see any substantial discontinuities, with 1944 seeing the largest change, equal to only 4 years of progress at the previous rate. Unfortunately, we were unable to find quantitative data prior to 1941, so we were only able to track progress for the three years leading up to the mass production of penicillin.
From our perspective, progress by 1943 may already have been affected by availability of penicillin that we do not know about, in which case we have no earlier trend to go by. However we note that the scale of annual reductions following penicillin is not larger than the increase seen in 1943, and not vastly larger than later annual variations, so the largest abrupt decrease from penicillin seems unlikely to have been large compared to the usual scale of variation.
US Deaths from syphilis
Data
We collected data from two graphs of historical US syphilis deaths and put it in this spreadsheet. The first is shown in Figure 2, and comes from Armstrong et al.’s 1999 report on infectious disease mortality in the United States.9 The authors collected it from historical mortality and population data from the CDC and public use mortality data tapes.10 We used an automatic figure data extraction tool to extract data from the figure.11 Mortality rates after the mid-60s are indistinguishable from zero in this figure, so we do not include them. Instead we include records of total US deaths from Peterman & Kidd, 201912, which we combine with US population data to get mortality rates between 1957 and 2015.

Figure 2: Syphilis mortality rate in the US during the 20th century.13
Figure 3: Syphilis mortality rate in the US during the 20th century, plotted on a log scale
Discontinuity Measurement
We calculate discontinuities in our spreadsheet, according to this methodology. There were no substantial discontinuities in progress for reducing syphilis deaths in the US during the time for which we have data. The largest positive deviation from a previous trend was a drop representing five years of progress in around 1940, two years before even enough ‘US penicillin’ was available to treat ten people.14
In sum, while deaths from syphilis rapidly declined around the 1940s, this progress was not discontinuous at the scale of years. And while penicillin seems likely to have helped in this decline, it did not yet exist to contribute to the most discontinuously fast progress in that trend (and that progress was still not rapid enough to count as a substantial discontinuity for this project).
Discussion of causes
The decline of syphilis mortality does not appear to be entirely from penicillin, since it is underway by 1940, just prior to the mass-production of penicillin. This is strange, so it is plausible that we misunderstand some aspect of the situation.
The only other factor we know about is US Surgeon General Thomas Parran’s launch of a national syphilis control campaign in 1938.15 Wikipedia also attributes some of the syphilis decline over the 19th and 20th centuries to decreasing virulence of the spirochete, but we don’t know of any reason for that to especially coincide with the 1940s decline.16
Effectiveness at treating syphilis
Even if penicillin’s effect on the US death rate from syphilis was gradual, we might expect this to be due to frictions like institutional inertia, rather than from gradual progress in the underlying technology. It might still be that penicillin was a radically better drug than its predecessors, when applied.
We briefly investigated whether penicillin might have represented discontinuous progress in effectiveness at curing syphilis, and conclude that it probably did not, because it does not appear to have been clearly better than its predecessor in terms of cure rates. In a 1962 review of treatment of ‘early’ syphilis17, Willcox writes that ‘a seronegativity-rate of 85 per cent. at 11 months had been achieved’ in 1944 after penicillin became the primary treatment for syphilis, but also says that the previously common treatment—arsenic and bismuth—was successful in more than 90% of cases in which it was carried out.18
Willcox explains that the major downsides of the earlier treatment were very high defection rates (with perhaps as few as a quarter of patients completing the treatment), and ‘serious toxic effects’.19 We have not checked that exactly the same notion of success is being used in these figures, have not assessed the reliability of this source, and do not know how important treatment for ‘early’ syphilis is relative to treatment for all syphilis, so it could still be that penicillin was a more effective treatment overall. However we did not investigate this further.
Inclusive costs of treatment
Penicillin apparently allowed most patients to receive a curative dose of medicine, whereas ‘arsenic and bismuth therapy’ achieved this for perhaps as few as a quarter of patients.20 If penicillin made an abrupt difference to syphilis treatment then, it seems likely to have been in terms of inclusive costs (which were partly reflected in willingness to be treated).
Qualitatively, the costs of treatment do seem to have been much lower. The time for treatment dropped from a year to around eight days.21 Our impression is that the side effects qualitatively reduced from horrible and sometimes deadly to apparently bearable.
However even if penicillin was a large improvement over its predecessors in absolute terms (which seems likely), it would be hard to make a clear case that it was large relative to previous progress in syphilis treatments, because recent progress was also incredible.
The ‘arsenic and bismuth therapy’ mentioned above, that preceded penicillin, seems to have been a combination of the arsenic-based drug salvarsan (arsphenamine) and similar drugs developed subsequently, with bismuth. 22 Salvarsan (arsphenamine) was considered such radical improvement over its own predecessors that it was known as the ‘magic bullet’, and won its discoverer Paul Erhlich a Nobel prize.23 A physician at the time describes24:

“Arsenobenzol, designated “606,” whatever the future may bring to justify the present enthusiasm, is now actually a more or less incredible advance in the treatment of syphilis and in many ways is superior to the old mercury – as valuable as this will continue to be – because of its eminently powerful and eminently rapid spirochaeticidal property.”

It is easy to see how salvarsan could be hugely costly to take, yet still represent large progress over earlier options, when we note that the common treatment prior to salvarsan was mercury,25 which had ‘terrible side effects’ including the death of many patients, characteristically took years, and was not obviously helpful.26
So at a glance penicillin doesn’t look to have been clearly discontinuous relative to the impressive recent trend, and measuring inclusive costs is hard to do finely enough to see less clear discontinuities. Thus evaluating these costs quantitatively will remain beyond the scope of this investigation at present. We tentatively guess that penicillin did not represent a large discontinuity in inclusive costs of syphilis treatment, though it did represent huge progress.
Conclusions
Penicillin probably made quick but not abrupt progress in reducing syphilis and syphilis mortality. Penicillin doesn’t appear to have been much more likely to cure a patient than earlier treatments, conditional on the treatment being carried out, but it penicillin treatment appears to have been around four times more likely to be carried out, due to lower costs. Qualitatively penicillin represented an important reduction in costs, but it is hard to evaluate this precisely or compare it with the longer term progress. It appears that as recently as 1910 another drug for syphilis also represented qualitatively huge progress in treatment, so it is unlikely that penicillin was a large discontinuity relative to past progress.
Notes

Penicillin and historic syphilis trends


Historic trends in the maximum superconducting temperature

The maximum superconducting temperature of any material up to 1993 contained four greater than 10-year discontinuities: A 14-year discontinuity with NbN in 1941, a 26-year discontinuity with LaBaCuO4 in 1986, a 140-year discontinuity with YBa2Cu3O7 in 1987, and a 10-year discontinuity with BiCaSrCu2O9 in 1987.
YBa2Cu3O7 superconductors seem to correspond to a marked change in the rate of progress of maximum superconducting temperature, from a rate of progress of .41 Kelvin per year to a rate of 5.7 Kelvin per year.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
Superconductors were discovered in 1911.1 Until 1986 the maximum temperature for superconducting behavior had gradually risen from around 4K to less than 30K (see figure 2 below). Theory at the time apparently predicted that 30K was an upper limit.2 In 1986 a new class of ceramics known as YBCO superconductors was discovered to allow superconducting behavior at higher temperatures: above 80K,3 and within seven years, above 130K.4
Figure 1: Levitation of a magnet above a superconductor5
Trends
Maximum temperature for superconducting behavior
We looked at data for the maximum temperature at which any material is known to have superconducting behavior.
Data
We found the following data in a figure from the University of Cambridge’s online learning materials course, DoITPoMS,6 and have verified most of it against other data sources (see our spreadsheet, where we also collected ‘Extended data’ to verify that these were indeed the record temperatures).
We display the original figure from DoITPoMS in Figure 2 below, followed by our figure, Figure 3, which includes the a more recent superconducting material, H2S.
Figure 2:  Maximum superconducting temperature by material over time through 2000, from the University of Cambridge’s online learning materials course, DoITPoMS,7
Figure 3: Maximum superconducting temperate by material over time through 2015
Discontinuity measurement
We modeled this data as linear within two different regimes, one up to LaBaCu04 in 1986, and another starting with 1986 until our last data point.8 Using previous rates from those trends, we calculated four greater than 10-year discontinuities (rounded), shown in the table below:9
YearTemperatureDiscontinuityMaterial194116 K14 yearsNbN198635 K26 yearsLaBaCuO4198793 K140 yearsYBa2Cu3O71987105 K10 yearsBiCaSrCu2O9
In addition to the size of this discontinuity in years, we have tabulated a number of other potentially relevant metrics here.10
Changes in the rate of progress
We note that there was a marked change in the rate of progress of maximum superconducting temperature with YBa2Cu3O7. The maximum superconducting temperature changed from a rate of progress of .41 Kelvin per year to a rate of 5.7 Kelvin per year.11
Notes


Historic trends in chess AI

The Elo rating of the best chess program measured by the Swedish Chess Computer Association did not contain any greater than 10-year discontinuities between 1984 and 2018. A four year discontinuity in 2008 was notable in the context of otherwise regular progress.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
The history of chess-playing computers is long and rich, partly because chess-playing ability has long been thought (by some) to be a sign of general intelligence.1 The first two ‘chess-playing machines’ were in fact fakes, with small human chess-players crouching inside.2 It was not until 1951 that a program was published (by Alan Turing) that could actually play the full game.3 There has been fairly regular progress since then.4
In 1997 IBM’s chess machine Deep Blue beat Gary Kasparov, world chess champion at the time, under standard tournament time controls.5 This was seen as particularly significant in light of the continued popular association between chess AI and general AI.6 The event marked the point at which chess AI became superhuman, and received substantial press coverage.7
The Swedish Chess Computer Association (SSDF) measures computer chess software performance by playing chess programs against one another on standard hardware.8
Figure 1: Deep Blue9
Trends
SSDF Elo Ratings
According to Wikipedia10:
The Swedish Chess Computer Association (Swedish: Svenska schackdatorföreningen, SSDF) is an organization that tests computer chess software by playing chess programs against one another and producing a rating list…The SSDF list is one of the only statistically significant measures of chess engine strength, especially compared to tournaments, because it incorporates the results of thousands of games played on standard hardware at tournament time controls. The list reports not only absolute rating, but also error bars, winning percentages, and recorded moves of played games.
Data
We took data from Wikipedia’s list of SSDF Ratings11 (which we have not verified) and added it to this spreadsheet. See Figure 2 below.
  Figure 2: Elo ratings of the best program on SSDF at the end of each year.
Discontinuity measurement
Looking at the data, we assume a linear trend in Elo.12 There are no discontinuities of 10 or more years.
Minor discontinuity
There is a four year discontinuity in 2008. While this is below the scale of interest for our discontinuous progress investigation,  it strikes us as notable in the context of otherwise very regular progress.13 We’ve tabulated a number of other potentially relevant metrics for this discontinuity in the ‘Notable discontinuities less than 10 years’ tab here.14
This jump appears to have been partially caused by the introduction of new hardware in the contest, as well as software progress.15
Notes

Historic trends in chess AI


Effect of Eli Whitney’s cotton gin on historic trends in cotton ginning

We estimate that Eli Whitney’s cotton gin represented a 10 to 25 year discontinuity in pounds of cotton ginned per person per day, in 1793. Two innovations in 1747 and 1788 look like discontinuities of over a thousand years each on this metric, but these could easily stem from our ignorance of such early developments. We tentatively doubt that Whitney’s gin represented a large discontinuity in the cost per value of cotton ginned, though it may have represented a moderate one.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
Cotton fibers grow around cotton seeds, which they need to be separated from before use. This can be done by hand, but since 500 C.E.,1 and plausibly prehistory, a variety of tools have aided in speeding up the process. 2
These tools are called ‘cotton gins’. Eli Whitney’s 1793 cotton gin was a particularly famous innovation, commonly credited with having vastly increased cotton’s profitability, fueling an otherwise diminishing demand for slave labor, and so substantially contributing to the American Civil War.3 Variants on Whitney’s gin are known as ‘saw gins’.4 (See Figure 1.) Cotton became more valuable than all other US exports combined during the antebellum era.5 Thus Whitney’s gin is a good contender for representing a discontinuity in innovation.
Our investigation draws heavily from Lakwete’s Inventing the Cotton Gin. Lakwete summarizes the situation surrounding Whitney’s invention as follows6:
The introduction of a new gin in 1794 was as unexpected as it was unprecedented. It was unexpected because the British textile industry had expanded from the sixteenth through the eighteenth centuries without a change in the ginning principle. Cotton producers had increased the acres they planted in cotton and planted new varieties to suit textile makers. The market attracted new producers who, like established planters, used roller gins to process their crops. Roller gins, whether hand-cranked in the Levant and India, or foot-, animal-, and inanimately powered in the Americas, provided adequate amounts of fiber with the qualities that textile makers wanted, namely length and cleanliness. All roller gins removed the fiber by pinching it off in bundles, preserving its length and orientation as grown. Random fragments of fractured seeds were picked out of the fiber before it was bagged and shipped. In 1788 Joseph Eve gave planters and merchants a machine that bridged the medieval and modern. It preserved the ancient roller principle but completed the appropriation of the ginner’s skill, as Arkwright’s frame had that of the spinner. Appropriation had proceeded in stages beginning with the single-roller gin that mechanized the thumb and finger pinching motion. The roller gin in turn appropriated the agility and strength needed to manipulate the single roller, while the foot gin freed both hands to supply seed cotton. The barrel gin used animal and water power, removing humans as a power source but retaining them as seed cotton suppliers. The self-feeding animal-, wind-, or water-powered Eve gin replaced each of the skilled tasks of the ginner with mechanical components. Nevertheless, Eli Whitney’s unprecedented gin filled a vacuum. While large merchants invested in barrel gins and large planter in the Eve gin, the majority continued to use the skill- and labor-intensive foot gin to gin fuzzy-seed short-staple cotton as well as the smooth-seed, Sea Island cotton. Barrel gins had not decreased the number of ginners and only marginally improved ginner productivity, and Eve’s complicated gin was notoriously finicky. Whitney ignored these modernizing gins and offered a replacement for the ubiquitous foot gin.
Figure 1: What appears to be a saw gin of some kind on display at the Eli Whitney Museum7
Trends
Pounds of cotton ginned per person-day
We are most interested in metrics that people were working to improve—in this case, perhaps ‘cost of producing a dollar’s worth of cotton’. Inclusive metrics are hard to measure however. Instead we have collected data on ‘pounds of cotton ginned per person per day’, which is simpler, often reported on, and probably a reasonable proxy. However, it departs from tracking the usefulness of a gin by ignoring several major factors:

Upfront costs: these presumably varied a lot, because a gin can for instance resemble a rolling pin and a board, or involve horses or steam power.8 Thus the gins with higher upfront costs are less useful than their cotton-per-person-day statistic would make them seem. In the mid-1860’s many farms still used foot gins, seemingly because Eve gins—while more efficiently producing high quality output—were expensive.9 Even if everything had the same upfront costs, the existence of upfront costs means that a gin which processes 200lb of cotton with two people per day would be better than one that processes 100lb of cotton with one person, so cotton/person-day still fails to match what we are interested in.
Variation in labor requirements: Some gins required especially skilled labor.10
Substitutes for people: some gins used people to power them and others used animals or water-power, along with a smaller number of people.11 This again makes for higher output per person, but at the cost of additional animals, that we are not accounting for.

Risks of injury: Some gins, particularly foot and barrel gins, were dangerous to operate.12

Types and quality of cotton ginned: Whitney’s gin produced degraded cotton fiber, relative to other gins available at the time.13 However, Whitney’s gin could process short staple cotton, an easier to grow strain which was previously hard to process.14 The cotton industry might adjust to different cotton over time, so that long-run differences in quality of outputs of different gins are smaller than initial differences. If so, we expect value produced by a new gin producing lower quality cotton to grow continuously over an extended period.

We do also investigate overall cost per value of cotton ginned later, but do not have such clear data for it (see section, ‘cost per value of cotton ginned’).
Data
We collected claims about cotton gin productivity in the time leading up to Whitney’s gin, and some after. Many but not all are from Angela Lakwete’s book, Inventing the Cotton Gin: Machine and Myth in Antebellum America.15 Our sources are mostly secondhand or thirdhand claims about nonspecific observations in the 1700s. We have the impression that claims in this space are not very reliable.16 We classified claims as ‘credible’ or not, but this is fairly ambiguous, and we would be unsurprised if some of the ‘credible’ claims turned out to be inaccurate, or the ‘non-credible’ ones were correct.
Our dataset of claims is here, and illustrated in Figures 2 – 5. Note that dates are those when a claim was made, not necessarily dates of the invention of the type of cotton gin in question. This is because invention dates are hard to find, and also because it seems likely that much improvement happened incrementally between distinct ‘inventions’ of new types. Nonetheless, this means that a report dated to a time could be from a gin that was built earlier.
Figure 2: Claimed cotton gin productivity, 1720 to modern day, coded by credibility and being records, and dated by when the claim was made (not necessarily when the gin was made). Claims that are both relatively credible and higher than previous relatively credible claims are few. The last credible best point before the modern day is an improved version of Whitney’s gin, two years after the original (the original features in the two high non-credible claims slightly earlier).
Figure 3: Historic claimed cotton gin productivity, all time (zoomed out version of Figure 2)
Figure 4: Zoom-in on credible best cotton gins (excluding modern era)
Discontinuity measurement
For measuring discontinuities, we treat past progress as exponential at each point, but entering a new exponential regime at the fourth point. We confine our investigation to credible records. Given these things, we find the improved Whitney gin to be a 23-year discontinuity over the previous record in this dataset. However the foot gin and Eve’s mill gin appear to be at least one-thousand year discontinuities each.17.
However our data has at least one key gap. Whitney’s original 1793 gin design was almost immediately copied and improved by many people, most notably Hodgen Holmes and Daniel Clark.18 The plausible productivity data we have appears to all be for these later variants, which we understand were non-negligibly better than Whitney’s original gin in some way.19 So we know that Whitney’s gin should be somewhat lower and two years earlier than our first data for Whitney-style gins. This means at most, Whitney’s original gin would be a 25 year discontinuity. If it accounted for even half of the progress since Eve’s mill gin, and we are not missing further innovations between the two, Whitney’s gin would still represent a 13 year discontinuity, and the later improved version would no longer account for a discontinuity of more than ten years.20 It seems likely to us that Whitney’s gin was at least this revolutionary so we think the Whitney gin probably represented a moderate (10-25 year) discontinuity in pounds of cotton ginned per day.
We are fairly uncertain about whether the two larger discontinuities earlier are real, or due to gaps in our data. We did attempt to collect data for these earlier times (rather than just prior to the Whitney gin), but seem very likely to be missing a lot.
In addition to the size of these discontinuities in years, we have tabulated a number of other potentially relevant metrics here.21
Changes in the rate of progress
Over the history of gin productivity, the average rate of progress became higher. It is unclear whether this happened at a particular point. In our data, it looks as though it happened with the foot gin, in 1747, and that progress went from around .05% per year to around 4% per year (see our spreadsheet, tab ‘credible record gin calculations’). However our data is too sparse and uncertain to draw firm conclusions from.
Cost per value of cotton ginned
As discussed above, pounds of cotton ginned per person-day is not a perfect proxy of the value of a cotton gin, and therefore presumably not exactly what cotton-gin users were aiming for. ‘Cost per value of cotton ginned’ seems closer, if we measure costs inclusively and average across various cotton ginning situations. We did not collect data on this, but can make some inferences about the shape of this trend—and in particular whether Whitney’s gin represented a discontinuity—from what we know about the pounds/person-day figures and other aspects of the situation.
Evidence from the trend in pounds of cotton ginned per person-day
We expect that the pounds of cotton ginned per person-day roughly approximates cost per value of cotton ginned, with the following adjustments that we know of:
Eve’s gin is worse on cost/value than on cotton/person-day because the latter metric doesn’t reflect its large upfront costs.Whitney’s gin may be worse on cost per value than it appears, because of its lower-quality cotton output.Whitney’s gin may be better than it appears, because it could handle short-staple cotton. However this value seems unlikely to have manifest immediately, since it presumably takes time for cotton users to adjust to a new material.Foot gins and barrel gins (e.g. Eve’s) were dangerous to operate, so are worse on cost/value than they appear.Foot gins apparently required especially skilled labor, so are worse on cost/value than they appear.Barrel gins and Eve gins often ran on non-human power-sources, so are worse on cost/value than they appear.22
These are several considerations in favor of Whitney’s gin representing more progress on cost/value than on cotton/person-day, and one against. However it is unclear to us whether the downside of lower quality cotton was larger than the other considerations combined, so the overall effect on the expected size of discontinuity from Whitney’s gin seems ambiguous, but probably in favor of larger.
Evidence from takeup of gins
The foot gin persisted for at least sixty years after Whitney’s invention.23 This suggests that Whitney’s gin wasn’t radically better on cost per value of cotton ginned than its predecessors, at least for some cotton producers.
On the other hand, apparently there was a rush to manufacture copies of Whitney’s gin, so much so that many mechanics became professional gin-makers, and most plantations had one of the new gins within five years. 24
This suggests that there were situations for which Whitney’s gin was substantially better than alternatives, and situations for which it was worse. This seems like weak evidence that on average across cotton ginning needs it was not radically better than precursors, though there might be narrower metrics we could define on which it was radically better.
Evidence from cotton production trends
If the Whitney gin made cotton much cheaper to process, we might expect cotton production at the time to sharply increase. Our impression is that this is a common story about what happened.25 However the data we could find on this, seemingly from a 1958 history of early US agriculture, suggests that cotton production was already growing rapidly, and continued on a similar trajectory after Whitney’s invention.26 See Figure 5.
This dataset begins in 1790, only a few years before Whitney’s invention. This is enough to see that the trend just before 1793 is much like the trend just after, however we can further verify this by looking at earlier cotton export figures (Figure 7).27 Cotton exports appear to closely match overall productivity where the trends overlap, and the pre-1790 export trend appears to be roughly continuous with the rest of the curve, at least if we ignore the aberrantly low 1790 figure.
Figure 5: historic cotton production (bales), probably from Gray et al 195828, see above for elaboration on source, data here.
Figure 6: Rough collected figures for US cotton exports and production over period from 1780 – 1830, data and sources here.
Figure 7: close up on relevant years from Figure 6
This does not preclude a large change in gin efficacy—perhaps there were other bottlenecks to cotton productivity, or it took time for the gains from Whitney’s gin to manifest in national productivity data. However it does cause us to doubt the story of Whitney’s gin being evidently responsible for massive growth in the cotton industry, which was a reason for suspecting the gin may have represented discontinuous progress. So this is some evidence against Whitney’s gin representing a large discontinuity in cost per value of cotton ginned.
Evidence from the 1879 evaluation of ginning technology
This 1879 evaluation of ginning technology reports on extensive measurement trials of different cotton gins. It was seemingly conducted to understand why Indian cotton production lagged behind American. The author says all methods for ginning cotton in India were primitive until recently; he hears that in some places it was done by hand as late as 1859.29 This is confusing because if Whitney’s gin really was much better in terms of cost-per-value than the alternatives, it would be surprising if sixty years later the alternatives were still in use. However many alternatives seem clearly more cost-effective than ginning by hand, so this seems like little evidence about Whitney’s gin in particular.
The outputs of the gins in the experiment seem different from (and usually higher than) the outputs for similarly named gins in our dataset. Which might be confusing, but we expect is because there are modest improvements to gin technology over time within particular classes of gin.
In sum, this evidence looks as though it might be informative, but we do not see it as such on consideration.
Evidence from historians
We have not thoroughly reviewed popular or academic opinions on the discontinuousness of the cotton gin, but our impression is that a common popular view is that Eli Whitney’s cotton gin was a discontinuous improvement over the state-of-the-art. On the other hand Dr Lakwete, author of Inventing the Cotton Gin—a book we found most helpful in this project, and that also won an award for being the best scholarly book published about the history of technology30—disagrees, actually explicitly saying it was continuous (though she may mean something different by this than we do):
Collapsing two hundred years of cotton production and roller gin use in North America in to the moment when Eli Whitney invented the toothed gin, Phineas Miller and Judge Johnson marked 1794 as a turning point in southern development. Before, southeners languished without an effective gin for short-staple cotton; afterwards, the cotton economy blossomed. Arguing for discontinuity, the idea allowed the visualization of a moment and a machine that separated the colonial past from the new republic. Continuity, however, marked the history of cotton and the gin in America. Continuity would characterize the first two decades of the nineteenth century, as saw-gin and roller gin makers competed for dominance in the expanding short-staple cotton market.Lakwete, Angela. Inventing the Cotton Gin: Machine and Myth in Antebellum America. Baltimore, MD: Johns Hopkins University Press, 2003. , 71.
Conclusions on cost per value of cotton ginned
On the earlier ‘pounds of cotton ginned per person per day’ metric, we estimated that Whitney’s gin was worth around 10-25 years of past progress. Various considerations suggested Whitney’s gin might have been a bigger deal for overall cost-effectiveness of ginning cotton than that calculation suggested, but the quality of cotton was lower. We took this in total as neutral to weakly favoring Whitney’s gin being better than it seemed. We then saw that the Whitney gin was taken up with enthusiasm by a subset of people needing to gin cotton, that it didn’t seem to recognizably affect the growth of US cotton production, and that at least one historian with particular expertise in this topic thinks that progress was relatively continuous. This does not particularly suggest to us that Whitney’s gin represented a large discontinuity in cost per value of cotton ginned, and seems like some evidence against.
Notes


Historic trends in flight airspeed records

Flight airspeed records between 1903 and 1976 contained one greater than 10-year discontinuity: a 19-year discontinuity corresponding to the Fairey Delta 2 flight in 1956.
The average annual growth in flight airspeed markedly increased with the Fairey Delta 2, from 16mph/year to 129mph/year.
Details
This case study is part of AI Impacts’ discontinuous progress investigation.
Background
Flight airspeed records are measured relative to particular classes of aircraft, with official rules defined by the Fédération Aéronautique Internationale (FAI). is “the highest airspeed attained by any aircraft of a particular class”.1
Trends
Flight airspeed records
Data
We took data from Wikipedia’s list of flight airspeed records2 (which we have not verified) and added it to this spreadsheet. We understand it to be fastest records across all classes of manned aircraft that are able to take off under their own power, but it is not well explained on the page. We included only official airspeed records. See Figure 1 below.
Figure 1: Flight airspeed records over time
Discontinuity measurement
We treat the data as linear, and once deem it to have begun a new trend, for the purpose of determining the past rate of progress. 3 We calculate the size of discontinuities in this spreadsheet.4 In 1956, there was a 19-year discontinuity in flight airspeed records with the Fairey Delta 2 flight.
We tabulated a number of other related metrics here.5
Figure 2: Fairey Delta 26, whose 1956 record represented a 19 year discontinuity.
Change in the growth rate
The average annual growth in flight airspeed markedly increased at around the time of the Fairey Delta 2. Airspeed records grew by an average of 16mph/year up until the one before Fairey Delta 2, whereas from that point until 1965 they grew by an average of 129mph/year.7
Notes


Comparison of naturally evolved and engineered solutions

This page describes a project that is in progress, and does not yet have results
We are comparing naturally evolved and engineered solutions to problems, to learn about regularities that might let us make inferences about artificial intelligence from what we know about naturally evolved intelligence.
Details
Motivation
Engineers and evolution have faced many similar design problems. For instance, the problem of designing an efficient flying machine. Another instance of a design problem that engineers and evolution have both worked on is designing intelligent machines. We hope that by looking at other instances of engineers and evolution working on similar problems, we will be able to learn more about how future AI systems will compare to evolved intelligences.
Methods
We will collect examples of optimization problems that engineers and evolution would perform better on if they could. Here are some candidate examples of such problems:
FlyingHoveringSwimmingRunningTraveling long distancesTraveling quicklyJumpingBalancingHeight of structurePiercingApplying compressive forceStrikingTensile strengthPumping bloodBreathingLiver functionDetecting lightRecording lightProducing lightDetecting soundRecording soundProducing soundHeat insulationDetermining chemical composition of a substanceDetecting chemical composition in the airAdhesivenessPicking heavy things upJoint activationElasticityToxicityExtracting energy from sunlightStoring energy
We will then collect the best solutions we can readily find to these design problems, made by human engineers and by evolution respectively, and quantitative data on their performances. We will try to collect this over time, for engineered solutions.
Analysis
We will use the data to answer the following questions for different design problems:
How long does it take engineers to half, match, double, triple, etc. the performance of evolution’s current best designs? What does the shape of engineers’ performance curve look like around the point where engineers’ solutions first match evolution’s? How efficient (in terms of performance per energy or mass used) are the first solutions that match evolution’s performance compared to evolution’s best solutions? How long does it take engineers to find a more efficient solution after finding an equally good solution in terms of absolute performance? From a design perspective, how similar are engineers’ first equally good solutions to evolution’s best solutions?
We will use patterns in the answers to these questions across technologies to make inferences about the answers for natural and artificial intelligence.
In general, the more similar the answers to these questions turn out to be across design problems, the more strongly we will expect the answers for problems addressed by future AI developments to fit the same patterns.
We expect to make the data publicly available, so that others can check our conclusions, investigate related questions, or use it in other investigations of technology and evolution.


Walsh 2017 survey

Toby Walsh surveyed hundreds of experts and non-experts in 2016 and found their median estimates for ‘when a computer might be able to carry out most human professions at least as well as a typical human’ were as follows:
Probability of HLMIGroup of survey respondentsAI expertsRobotics expertsNon-experts10%20352033202650%20612065203990%210921182060
Details
Toby Walsh, professor of AI at the University of New South Wales and Technical University of Berlin, conducted a poll of AI experts, robotics experts, and non-experts from late January to early February 2017. The survey focused on the potential automation of various occupations and the arrival of high-level machine intelligence (HLMI).
Survey respondents
There were 849 total survey respondents composing three separate groups: AI experts, robotics experts, and non-experts.
The AI experts consisted of 200 authors from two AI conferences: the 2015 meeting of the Association for the Advancement of AI (AAAI) and the 2011 International Joint Conference on AI (IJCAI).
The robotics experts consisted of 101 individuals who were either Fellows of the Institute for Electrical and Electronics Engineers (IEEE) Robotics & Automation Society or authors from the 2016 meeting of the IEEE Conference on Robotics & Automation (ICRA).
The non-experts consisted of 548 readers of an article about AI on the website The Conversation. While it seems data on their possible expertise in AI or robotics was not collected, Walsh writes that “it is reasonable to suppose that most are not experts in AI & robotics, and that they are unlikely to be publishing in the top venues in AI and robotics like IJCAI, AAAI or ICRA” (p. 635). Some additional demographic data was collected and reported (for this survey group only):
Geographic distribution: 36% Australia, 29% United States, 7% United Kingdom, 4% Canada, and 24% rest of the worldEducation: 85% have an undergraduate degree or higherAge: >33% are 34 or under, 59% are under 44, and 11% are 65 or olderEmployment status: >66% are employed and 25% are in or about to enter higher educationIncome: 40% reported an annual income of >$100,000
Classifying occupations at risk of automation
The first seven survey questions (out of eight total) asked respondents to classify occupations as either at risk of automation in the next two decades or not (binary response). For each occupation, respondents were provided with information about the work involved and skills required. There were 70 total occupations, which came from a previous study that had used a machine learning (ML) classifier to rank them in terms of their risk for automation. These rankings were then used in the present survey: Each question had respondents classify 10 occupations, starting with the five most likely and five least likely at risk of automation according to the ML classifier. This continued through subsequent questions until respondents classified all 70 occupations.
Arrival of high-level machine intelligence (HLMI)
The last survey question asked by what year there would be a 10%, 50%, and 90% chance of HLMI, which was defined as “when a computer might be able to carry out most human professions at least as well as a typical human” (p. 634). For each probability respondents chose from among eight options: 2025, 2030, 2040, 2050, 2075, 2100, After 2100, and Never. Median responses were calculated by interpolating the cumulative distribution function between the two nearest dates.
Results
Probability of when (in years) HLMI will arrive
Table 1 below summarized the median responses and is reproduced here for convenience.
Table 1
Probability of HLMIGroup of survey respondentsAI expertsRobotics expertsNon-experts10%20352033202650%20612065203990%210921182060
Figures 1-3 below show the cumulative distribution functions (CDFs) for 10%, 50%, and 90% probability of HLMI (respectively) at different years.

Figure 1

Figure 2

Figure 3
Occupations at risk of automation
Table 2 below contains descriptive statistics about the number of occupations (out of 70 total) classified as being at risk of automation in the next two decades. Confidence intervals (last column) are at the 95% level. It is unclear why the sample size for Non-experts is listed as 473 when earlier in the article the number reported is 548.
Table 2

The difference in means between the Robotics (29.0) and AI experts (31.1) was not statistically significant (two-sided t-test, p = 0.096), while the differences in means between both expert groups and the non-expert group (36.5) separately were significant (two-sided t-test, both p’s < 0.0001).
Table 3 below lists some of the largest differences in the proportion of experts (AI and robotics combined) compared to non-experts who classified occupations as at risk for automation.
Table 3
OccupationProportion of respondents predicting risk for automationExpertsNon-expertsEconomist12%39%Electrical engineer6%33%Technical writer31%54%Civil engineer6%30%
Figure 4 below shows that respondents who predicted that HLMI would arrive earlier also classified more occupations as being at risk of automation (and vice versa).

Figure 4

Walsh 2017 survey


Conversation with Adam Gleave

AI Impacts talked to AI safety researcher Adam Gleave about his views on AI risk. With his permission, we have transcribed this interview.
Participants
Adam Gleave — PhD student at the Center for Human-Compatible AI, UC BerkeleyAsya Bergal – AI ImpactsRobert Long – AI Impacts
Summary
We spoke with Adam Gleave on August 27, 2019. Here is a brief summary of that conversation:
Gleave gives a number of reasons why it’s worth working on AI safety:It seems like the AI research community currently isn’t paying enough attention to building safe, reliable systems.There are several unsolved technical problems that could plausibly occur in AI systems without much advance notice.A few additional people working on safety may be extremely high leverage, especially if they can push the rest of the AI research community to pay more attention to important problems.Gleave thinks there’s a ~10% chance that AI safety is very hard in the way that MIRI would argue, a ~20-30% chance that AI safety will almost certainly be solved by default, and a remaining ~60-70% chance that what we’re working on actually has some impact.Here are the reasons for Gleave’s beliefs, weighted by how much they factor into his holistic viewpoint:40%: The traditional arguments for risks from AI are unconvincing:Traditional arguments often make an unexplained leap from having superintelligent AIs to superintelligent AIs being catastrophically bad.It’s unlikely that AI systems not designed from mathematical principles are going to inherently be unsafe.They’re long chains of heuristic reasoning, with little empirical validation.Outside view: most fears about technology have been misplaced.20%: The AI research community will solve the AI safety problem naturally.20%: AI researchers will be more interested in AI safety when the problems are nearer.10%: The hard, MIRI version of the AI safety problem is not very compelling.10%: AI safety problems that seem hard now will be easier to solve once we have more sophisticated ML.Fast takeoff defined as “GDP will double in 6 months before it doubles in 24 months” is plausible, though Gleave still leans towards slow takeoff.Gleave thinks discontinuous progress in AI is extremely unlikely:There is unlikely to be a sudden important insight dropped into place, since AI has empirically progressed more by accumulation of lots of bags and tricks and compute.There isn’t going to be a sudden influx of compute in the near future, since well-funded organizations are currently already spending billions of dollars to optimize it.If we train impressive systems, we will likely train other systems beforehand that are almost as capable.Given discontinuous progress, the most likely story is that we combine many narrow AI systems in a way where the integrated whole is much better than half of them.Gleave guesses a ~10-20% chance that AGI technology will only be a small difference away from current techniques, and a ~50% chance that AGI technology will be easily comprehensible to current AI researchers:There are fairly serious roadblocks in current techniques right now, e.g. memory, transfer learning, Sim2Real, sample inefficiency.Deep learning is slowing down compared to 2012 – 2013:Much of the new progress is going to different domains, e.g. deep RL instead of supervised deep learning.Computationally expensive algorithms will likely hit limits without new insights.Though it seems possible that in fact progress will come from more computationally efficient algorithms.Outside view, we’ve had lots of different techniques for AI over time, so it would be surprising is the current one is the right one for AGI.Pushing more towards current techniques getting to AGI, from an economic point of view, there is a lot of money going into companies whose current mission is to build AGI.Conditional on advanced AI technology being created, Gleave gives a 60-70% chance that it will pose a significant risk of harm without additional safety efforts.Gleave thinks that best case, we drive it down to 20 – 10%, median case, we drive it down to 40 – 30%. A lot of his uncertainty comes from how difficult the problem is.Gleave thinks he could see evidence that could push him in either direction in terms of how likely AI is to be safe:Evidence that would cause Gleave to think AI is less likely to be safe:Evidence that thorny but speculative technical problems, like inner optimizers, exist.Seeing more arms race dynamics, e.g. between U.S. and China.Seeing major catastrophes involving AI, though they would also cause people to pay more attention to risks from AI.Hearing more solid arguments for AI risk.Evidence that would cause Gleave to think AI is more likely to be safe:Seeing AI researchers spontaneously focus on relevant problems would make Gleave think that AI is less risky.Getting evidence that AGI was going to take longer to develop.Gleave is concerned that he doesn’t understand why members of the safety community come to widely different conclusions when it comes to AI safety.Gleave thinks a potentially important question is the extent to which we can successfully influence field building within AI safety.
This transcript has been lightly edited for concision and clarity.
Transcript
Asya Bergal: We have a bunch of questions, sort of around the issue of– basically, we’ve been talking to people who are more optimistic than a lot of people in the community about AI. The proposition we’ve been asking people to explain their reasoning about is, ‘Is it valuable for people to be expending significant effort doing work that purports to reduce the risk from advanced artificial intelligence?’ To start with, I’d be curious for you to give a brief summary of what your take on that question is, and what your reasoning is.
Adam Gleave: Yeah, sure. The short answer is, yes, I think it’s worth people spending a lot of effort on this, at the margins, it’s still in absolute terms quite a small number. Obviously it depends a bit whether you’re talking about diverting resources of people who are already really dedicated to having a high impact, versus having your median AI researchers work more on safety related things. Maybe you think the median AI researcher isn’t trying to optimize for impact anyway, so the opportunity cost might be lower. The case I see from reducing the risk of AI is maybe weaker than some people in the community, but I think it’s still overall very strong.
The goal of AI as a field is still to build artificial general intelligence, or human-level AI. If we’re successful in that, it does seem like it’s going to be an extremely transformative technology. There doesn’t seem to be any roadblock that would prevent us from eventually reaching that goal. The path to that, the timeline is quite murky, but that alone seems like a pretty strong signal for ‘oh, there should be some people looking at this and being aware of what’s going on.’
And then, if I look at the state of the art in AI, there’s a number of somewhat worrying trends. We seem to be quite good at getting very powerful superhuman systems in narrow domains when we can specify the objective that we want quite precisely. So AlphaStar, AlphaGo, OpenAI Five, these systems are very much lacking in robustness, so you have some quite surprising failure modes. Mostly we see adversarial examples in image classifiers, but some of these RL systems also have somewhat surprising failure modes. This seems to me like an area the AI research community isn’t paying much attention to, and I feel like it’s almost gotten obsessed with producing flashy results rather than necessarily doing good rigorous science and engineering. That seems like quite a worrying trend if you extrapolate it out, because some other engineering disciplines are much more focused on building reliable systems, so I more trust them to get that right by default.
Even in something like aeronautical engineering where safety standards are very high, there are still accidents in initial systems. But because we don’t even have that focus, it doesn’t seem like the AI research community is going to put that much focus on building safe, reliable systems until they’re facing really strong external or commercial pressures to do so. Autonomous vehicles do have a reasonably good safety track record, but that’s somewhere where it’s very obvious what the risks are. So that’s kinda the sociological argument, I guess, for why I don’t think that the AI research community is going to solve all of the safety problems as far ahead of time as I would like.
And then, there’s also a lot of very thorny technical problems that do seem like they’re going to need to be solved at some point before AGI. How do we get some information about what humans actually want? I’m a bit hesitant to use this phrase ‘value learning’ because you could plausibly do this just by imitation learning as well. But there needs to be some way of getting information from humans into the system, you can’t just derive it from first principles, we still don’t have a good way of doing that.
There’s lots of more speculative problems, e.g. inner optimizers. I’m not sure if these problems are necessarily going to be real or cause issues, but it’s not something that we– we’ve not ruled it in or out. So there’s enough plausible technical problems that could occur and we’re not necessarily going to get that much advance notice of, that it seems worrying to just charge ahead without looking into this.
And then to caveat all this, I do think the AI community does care about producing useful technology. We’ve already seen some backlashes against autonomous weapons. People do want to do good science. And when the issues are obvious, there’s going to be a huge amount of focus on them. And it also seems like some of the problems might not actually be that hard to solve. So I am reasonably optimistic that in the default case of there’s no safety community really, things will still work out okay, but it also seems like the risk is large enough that just having a few people working on it can be extremely high leverage, especially if you can push the rest of the AI research community to pay a bit more attention to these problems.
Does that answer that question?
Asya Bergal: Yeah, it totally does.
Robert Long: Could you say a little bit more about why you think you might be more optimistic than other people in the safety community?
Adam Gleave: Yeah, I guess one big reason is that I’m still not fully convinced by a lot of the arguments for risks from AI. I think they are compelling heuristic arguments, meaning it’s worth me working on this, but it’s not compelling enough for me to think ‘oh, this is definitely a watertight case’.
I think the common area where I just don’t really follow the arguments is when you say, ‘oh, you have this superintelligent AI’. Let’s suppose we get to that, that’s already kind of a big leap of faith. And then if it’s not aligned, humans will die. It seems like there’s just a bit of a jump here that no one’s really filled in.
In particular it seems like sure, if you have something sufficiently capable, both in terms of intelligence and also access to other resources, it could destroy humanity. But it doesn’t just have to be smarter than an individual human, it has to be smarter than all of humanity potentially trying to work to combat this. And humanity will have a lot of inside knowledge about how this AI system works. And it’s also starting from a potentially weakened position in that it doesn’t already have legal protection, property ownership, all these other things.
I can certainly imagine there being scenarios unfolding where this is a problem, so maybe you actually give an AI system a lot of power, or it just becomes so, so much more capable than humans that it really is able to outsmart all of us, or it might just be quite easy to kill everyone. Maybe civilization is just much more fragile than we think. Maybe there are some quite easy bio ex-risks or nanotech that you could reason about from first principles. If it turned out that a malevolent but very smart human could kill all of humanity, then I would be more worried about the AI problem, but then maybe we should also be working on the human x-risk problem. So that’s one area that I’m a bit skeptical about, though maybe flushing that argument out more is bad for info-hazard reasons.
Then the other thing is I guess I feel like there’s a distribution of how difficult the AI safety problem is going to be. So there’s one world where anything that is not designed from mathematical principles is just going to be unsafe– there are going to be failure modes we haven’t considered, these failure modes are only going to arise when the system is smart enough to hurt you, and the system is going to be actively trying to deceive you. So this is I think, maybe a bit of a caricature, but I think this is roughly MIRI’s viewpoint. I think this is a productive viewpoint to inhabit when you’re trying to identify problems, but I think it’s probably not the world we actually live in. If you can solve that version, great, but it seems like a lot of the failure modes that are going to occur with advanced AI systems you’re going to see signs of earlier, especially if you’re actually looking out for them.
I don’t see much reason for AI progress to be discontinuous in particular. So there’s a lot of empirical records you could bring to bear on this, and it also seems like a lot of commercially valuable interesting research applications are going to require solving some of these problems. You’ve already seen this with value learning, that people are beginning to realize that there’s a limitation to what we can just write a reward function down for, and there’s been a lot more focus on imitation learning recently. Obviously people are solving much narrower versions of what the safety community cares about, but as AI progresses, they’re going to work on broader and broader versions of these problems.
I guess the general skepticism I have with the arguments, is, a lot of them take the form of ‘oh, there’s this problem that we need to solve and we have no idea how to solve it,’ but forget that we only need to solve that problem once we have all this other treasure trove of AI techniques that we can bring to bear on the problem. It seems plausible that this very strong unsupervised learning is going to do a lot of heavy lifting for us, maybe it’s going to give us a human ontology, it’s going to give us quite a good inductive bias for learning values, and so on. So there’s just a lot of things that might seem a lot stickier than they actually are in practice.
And then, I also have optimism that yes, the AI research community is going to try to solve these problems. It’s not like people are just completely disinterested in whether their systems cause harm, it’s just that right now, it seems to a lot of people very premature to work on this. There’s a sense of ‘how much good can we do now, where nearer to the time there’s going to just be naturally 100s of times more people working on the problem?’. I think there is still value you can do now, in laying the foundations of the field, but that maybe gives me a bit of a different perspective in terms of thinking, ‘What can we do that’s going to be useful to people in the future, who are going to be aware of this problem?’ versus ‘How can I solve all the problems now, and build a separate AI safety community?’.
I guess there’s also the outside view of just, people have been worried about a lot of new technology in the past, and most of the time it’s worked out fine. I’m not that compelled by this. I think there are real reasons to think that AI is going to be quite different. I guess there’s also just the outside view of, if you don’t know how hard a problem is, you should put a probability distribution over it and have quite a lot of uncertainty, and right now we don’t have that much information about how hard the AI safety problem is. Some problems seem to be pretty tractable, some problems seem to be intractable, but we don’t know if they actually need to be solved or not.
So, decent chance– I think I put a reasonable probability, like 10% probability, on the hard-mode MIRI version of the world being true. In which case, I think there’s probably nothing we can do. And I also put a significant probability, 20-30%, on AI safety basically not needing to be solved, we’ll just solve it by default unless we’re completely completely careless. And then there’s this big chunk of probability mass in the middle where maybe what we’re working on will actually have an impact, and obviously it’s hard to know whether at the margin, you’re going to be changing the outcome.
Asya Bergal: I’m curious– I think a lot of people we’ve talked to, some people have said somewhat similar things to what you said. And I think there’s two classic axes on which peoples’ opinions differ. One is this slow takeoff, fast takeoff proposition. The other is whether they think something that looks like current methods is likely to lead to AGI. I’m curious on your take on both those questions.
Adam Gleave: Yeah, sure. So, for slow vs. fast takeoff, I feel like I need to define the terms for people who use them in slightly different ways. I don’t expect there to be a discontinuity, in the sense of, we just see this sudden jump. But I wouldn’t be that surprised if there was exponential growth and quite a high growth rate. I think Paul defines fast takeoff as, GDP will double in six months before it doubles in 24 months. I’m probably mangling that but it was something like that. I think that scenario of fast takeoff seems plausible to me. I probably am still leaning slightly more towards the slow takeoff scenario, but it seems like fast takeoff will be plausible in terms of very fast exponential growth.
I think a lot of the case for the discontinuous progress argument falls on there being sudden insight that dropped into place, and it doesn’t seem to me like that’s what’s happening in AI, it’s more just a cumulation of lots of bags of tricks and a lot of compute. I also don’t see there being bags of compute falling out of the sky. Maybe if there was another AI winter, leading to a hardware overhang, then you might see sudden progress when AI gets funding again. But right now a lot of very well-funded organizations are spending billions of dollars on compute, including developing new application-specific integrated circuits for AI, so we’re going to be very close to the physical limits there anyway.
Probably the strongest case I see for discontinuities are the discontinuities you see when you’re training systems. But I just don’t think that’s going to be strong enough, because you’ll train other systems before that’ll be almost as capable. I guess we do see sometimes cases where one technique lets you solve a new class of problems.
Maybe you could see something where you get increasingly capable narrow systems, and there’s not a discontinuity overall, you already had very strong narrow AI. But eventually you just have so many narrow AI systems that they can basically do everything, and maybe you get to a stage where the integrated whole of those is much stronger than if you just had half of them, let’s say. I guess this is sort of the comprehensive AI services model. But again that seems a bit unlikely to me, because most of the time you can probably outsource some other chunks to humans if you really needed to. But yeah, I think it’s a bit more plausible than some of the other stories.
And then, in terms of whether I think current techniques are likely to get us to human-level AI– I guess I put significant probability mass on that depending on how narrowly you define it. One fuzzy definition is that a PhD thesis describing AGI being something that a typical AI researcher today could read and understand without too much work. Under this definition I’d assign 40 – 50%. And that could still include introducing quite a lot of new techniques, right, but just– I mean plausibly I think something based on deep learning, deep RL, you could describe to someone in the 1970s in a PhD thesis and they’d still understand it. But it’s just showing you, it’s not that much real theory that was developed, it was applying some pretty simple algorithms and a lot of compute in the right way. Which implies no huge new theoretical insights.
But if we’re defining it more narrowly, only allowing small variants of current techniques, I think that’s much less likely to lead to AGI: around 10-20%. I think that case is almost synonymous with the argument that you just need more compute, because it seems like there are so many things right now that we really cannot do: we still don’t have great solutions to memory, we still can’t really do transfer learning, Sim2Real just barely works sometimes. We’re still extremely sample inefficient. It just feels like all of those problems are going to require quite a lot of research in themselves. I can’t see there being one simple trick that would solve all of them. But maybe, current algorithms if you gave them 10000x compute would do a lot better on these, that is somewhat plausible.
And yeah, I do put fairly significant probability, 50%, on it being something that is kind of radically different. And I guess there’s a couple of reasons for that. One is, just trying to extrapolate progress forward, it does seem like there are some fairly serious roadblocks. Deep learning is slowing down in terms of, it’s not hitting as many big achievements as it was in the past. And also just AI has had many kinds of fads over time, right. We’ve had good old-fashioned AI, symbolic AI, we had expert systems, we had Bayesianism. It would be sort of surprising that the current method is the right one.
I don’t find that people are focusing on these techniques is necessarily particularly strong evidence that these systems are going to lead us to AGI. First, many researchers are not focused on AGI, and you can probably get useful applications out of current techniques. Second, AI research seems like it can be quite fashion driven. Obviously, there are organizations whose mission is to build AGI who are working within the current paradigm. And I think it is probably still the best bet, of the things that we know, but I still think it’s a bet that’s reasonably unlikely to pay off.
Does that answer your question?
Asya Bergal: Yeah.
Robert Long: Just on that last bit, you said– I might just be mixing up the different definitions you had and your different credences in those– but in the end there you said that’s a bet that you think is reasonably unlikely to pay off, but you’d also said 50% that it’s something radically different, so how– I think I was just confusing which ones you were on.
Adam Gleave: Right. So, I guess these definitions are all quite fuzzy, but I was saying 10-20% that something that is only a small difference away from current techniques would build AGI, and 50% that AGI was going to be comprehensible to us. I guess the distinction I’m trying to draw is the narrow one, which I give 10-20% credence, is we basically already have the right algorithms and we just need a few tricks and more compute. And the other more expansive definition, which I give 40-50% credence to, is allows for completely different algorithms, but excludes any deep theoretical insight akin to a whole new field of mathematics. So we might not be using back propagation any longer, we might not be using gradient descent, but it’ll be something similar — like the difference between gradient descent and evolutionary algorithms.
There’s a separate question of, if you’re trying to build AGI right now, where should you be investing your resources? Should you be trying to come up with a completely new novel theory, or should you be trying to scale up current techniques? And I think it’s plausible that you should just be trying to scale up techniques and figure out if we can push them forward, because trying to come up with a completely new way of doing AI is also very challenging, right. It’s not really a sort of insight you can force.
Asya Bergal: You kind of covered this earlier– and maybe you even said the exact number, so I’m sorry if this is a repeat. But one thing we’ve been asking people is the credence that without additional intervention– so imagining a world where EA wasn’t pushing for AI safety, and there wasn’t this separate AI safety movement outside of the AI research community, imagining that world. In that world, what is the chance that advanced artificial intelligence poses a significant risk of harm?
Adam Gleave: The chance it does cause a significant risk of harm?
Asya Bergal: Yeah, that’s right.
Adam Gleave: Conditional on advanced artificial intelligence being created, I think 60, 70%. I have a much harder time giving an unconditional probability, because there are other things that could cause humanity to stop developing AI. Is a conditional probability good enough, or do you want me to give an unconditional one?
Asya Bergal: No, I think the conditional one is what we’re looking for.
Robert Long: Do you have a hunch about how much we can expect dedicated efforts to drive down that probability? That is, the EA-focused AI safety efforts.
Adam Gleave: I think the best case is, you drive it down to 20 – 10%. I’m kind of picturing a lot of this uncertainty coming from just, how hard is the problem technically? And if we do inhabit this really hard version where you have to solve all of the problems perfectly and you have to have a formally verified AI system, I just don’t think we’re going to do that in time. You’d have to solve a very hard coordination problem to stop people developing AI without those safety checks. It seems like a very expensive process, developing safe AI.
I guess the median case, where the AI safety community just sort of grows at its current pace, I think maybe that gets it down to 40 – 30%? But I have a lot of uncertainty in these numbers.
Asya Bergal: Another question, going back to original statements for why you believe this– do you think there’s plausible concrete evidence that we could get or are likely to get that would change your views on this one direction or the other?
Adam Gleave: Yeah, so, seeing evidence of some of the more thorny but currently quite speculative technical problems, like inner optimizers, would make me update towards, ‘oh, this is just a really hard technical problem, and unless we really work hard on this, the default outcome is definitely going to be bad’. Right now, no one’s demonstrated an inner optimizer existing, it’s just a sort of theoretical problem. This is a bit of an unfair thing to ask in some sense, in that the whole reason that people are worried about this is that it’s only a problem with very advanced AI systems. Maybe I’m asking for evidence that can’t be provided. But relative to many other people, I am unconvinced by heuristic arguments appealing just to mathematical intuitions. I’m much more convinced either by very solid theoretical arguments that are proof-based, or by empirical evidence.
Another thing that would update me in a positive direction, as in AI seems less risky, would be seeing more AI researchers spontaneously focus on some relevant problems. There’s already, I guess this is a bit of a tangent, but I think maybe– people tend to conceive as the AI safety community as people who would identify as AI safety researchers. But I think the vast majority of AI safety research work is happening by people who have never heard of AI safety, but they have been working on related problems. This is useful to me all of the time. I think where we could plausibly end up having a lot more of this work happening without AI safety ever really becoming a thing is people realizing ‘oh, I want my robot to do this thing and I have a really hard time making it do that, let’s come up with a new imitation learning technique’.
But yeah, other things that could update me positively… I guess, AI seeming like a harder problem, as in, it seems like AI, general artificial intelligence is further away, that would probably update me in a positive direction. It’s not obvious but I think generally all else being equal, longer timelines is going to generally have more time to diagnose problems. And also it seems like the current set of AI techniques — deep learning and very data-driven approaches — are particularly difficult to analyze or prove anything about, so some other paradigm is probably going to be better, if possible.
Other things that would make me scared would be more arms race dynamics. It’s been very sad to me what we’re seeing with China – U.S. arms race dynamics around AI, especially since it doesn’t even seem like there is much direct competition, but that meme is still being pushed for political reasons.
Any actual major catastrophes involving AI would make me think it’s more risky, although it would also make people pay more attention to AI risk, so I guess it’s not obvious what direction it would push overall. But it certainly would make me think that there’s a bit more technical risk.
I’m trying to think if there’s anything else that would make more pessimistic. I guess just more solid arguments for AI safety, because a lot of my skepticism is coming from there’s just this very unlikely sounding set of ideas, and there are just heuristic arguments that I’m convinced enough by to work on the problem, but not convinced by enough to say, this is definitely going to happen. And if there was a way to patch some of the holes in those arguments, then I probably would be more convinced as well.
Robert Long: Can I ask you a little bit more about evidence for or against AGI being a certain distance away? You mentioned that as evidence that would change your mind. What sort of evidence do you have in mind?
Adam Gleave: Sure, so I guess a lot of the short timelines scenarios basically are coming from current ML techniques scaling to AGI, with just a bit more compute. So, watching for if those milestones are being achieved at the rate I was expecting, or slower.
This is a little bit hard to crystallize, but I would say right now it seems like the rate of progress is slowing down compared to something like 2012, 2013. And interestingly, I think a lot of the more interesting progress has come from, I guess, from going to different domains. So we’ve seen maybe a little bit more progress happening in deep RL compared to supervised deep learning. And the optimistic thing is to say, well, that’s because we’ve solved supervised learning, but we haven’t really. We’ve got superhuman performance on ImageNet, but not on real images that you just take on your mobile phone. And it’s still very sample inefficient, we can’t do few-shot learning well. Sometimes it seems like there’s a lack of interest on the part of the research community in solving some of these problems. I think it’s partly because no one has a solid angle of attack on solving these problems.
Similarly, while some of the recent progress in deep RL has been very exciting, it seems to have some limits. For example, AlphaStar and OpenAI Five both involved scaling up self-play and population based training. These were hugely computationally expensive, and that was where a lot of the scaling was coming from. So while there have been algorithmic improvements, I don’t see how you get this working in much more complicated environments without either huge additional compute or some major insights. These are things that are pushing me towards thinking deep learning will not continue to scale, and therefore very short timelines are unlikely.
Something that would update me towards shorter timelines would be if  something that I thought was impossible turns out to be very easy. So OpenAI Five did update me positively, because I just didn’t think PPO was going to work well in Dota and it turns out that it does if you have enough compute. I don’t think it updated me that strongly towards short timelines, because it did need a lot of compute, and if you scale it to a more complex game you’re going to have exponential scaling. But it did make me think, well, maybe there isn’t a deep insight required, maybe this is going to be much more about finding more computationally efficient algorithms rather than lots of novel insights.
I guess there’s also sort of economic factors– I mention mostly because I often see people neglecting them. One thing that makes me bullish on short timelines is that, there’s some very well-resourced companies whose mission is to build AGI. OpenAI just raised a billion, DeepMind is spending considerable resources. As long as this continues, it’s going to be a real accelerator. But that could go away: if AI doesn’t start making people money, I expect another AI winter.
Robert Long: One thing we’re asking people, and again I think you’ve actually already given us a pretty good sense of this, is just a relative weighting of different considerations. And as I say that, you actually have already been tagging this. But just to half review, from what I’ve scrawled down. A lot of different considerations in your relative optimism are: cases for AI as an x-risk being not as watertight as you’d like them, arguments for failure modes being the default and really hard, not being sold on those arguments, ideas that these problems might become easier to solve the closer we get to AGI when we have more powerful techniques, and then the general hope that people will try to solve them as we get closer to AI. Yeah, I think those were at least some of the main considerations I got. How strong relatively are those considerations in your reasoning?
Adam Gleave: I’m going to quote numbers that may not add up to 100, so we’ll have to normalize it at the end. I think the skepticism surrounding AI x-risk arguments is probably the strongest consideration, so I would put maybe 40% of my weight on that. This is because the outside view is quite strong to me, so if you talk about this very big problem that there’s not much concrete evidence for, then I’m going to be reasonably optimistic that actually we’re wrong and there isn’t a big problem.
The second most important thing to me is the AI research community solving this naturally. We’re already seeing signs of a set of people beginning to work on related problems, and I see this continuing. So I’m putting 20% of my weight on that.
And then, the hard version of AI safety not seeming very likely to me, I think that’s 10% of the weight. This seems reasonably important if I buy into the AI safety argument in general, because that makes a big difference in terms of how tractable these problems are. What were the other considerations you listed?
Robert Long: Two of them might be so related that you already covered them, but I had distinguished between the problems getting easier the closer we get, and people working more on them the closer we get.
Adam Gleave: Yeah, that makes sense. I think I don’t put that much weight on the problems getting easier. Or I don’t directly put weight on it, maybe it’s just rolled into my skepticism surrounding AI safety arguments, because I’m going to naturally find an argument a bit uncompelling if you say ‘we don’t know how to properly model human preferences’. I’m going to say, ‘Well, we don’t know how to properly do lots of things humans can do right now’. So everything needs to be relative to our capabilities. Whereas I find arguments of the form ‘we can solve problems that humans can’t solve, but only when we know how to specify what those problems are’, that seems more compelling, that’s talking about a relative strength between ability to optimize vs. ability to specify objectives. Obviously that’s not the only AI safety problem, but it’s a problem.
So yeah, I think I’m putting a lot of the weight on people paying more attention to these problems over time, so that’s probably actually 15 – 20% of my weight. And then I’ll put 5% on the problems getting easier and then some residual probability mass on things I haven’t thought about or haven’t mentioned in this conversation.
Robert Long: Is there anything you wish we had asked that you would like to talk about?
Adam Gleave: I guess, I don’t know if this is really useful, but I do wish I had a better sense of what other people in the safety community and outside of it actually thought and why they were working on it, so I really appreciate you guys doing these interviews because it’s useful to me as well. I am generally a bit concerned about lots of people coming to lots of different conclusions regarding how pessimistic we should be, regarding timelines, regarding the right research agenda.
I think disagreement can be healthy because it’s good to explore different areas. The ideal thing would be for us to all converge to some common probability distribution and we decide we’re going to work on different areas. But it’s very hard psychologically to do this, to say, ‘okay, I’m going to be the person working on this area that I think isn’t very promising because at the margin it’s good’– people don’t work like that. It’s better if people think, ‘oh, I am working on the best thing, under my beliefs’. So having some diversity of beliefs is good. But it bothers me that I don’t know why people have come to different conclusions to me. If I understood why they disagree, I’d be happier at least.
I’m trying to think if there’s anything else that’s relevant… yeah, so I guess another, this is merely just a question for you guys to maybe think about, is, I’m still unsure about how valuable field-building should be. And in particular, to what extent AI safety researchers should be working on this. It seems like a lot of reasons why I was optimistic assume the the AI research community is going to solve some of these problems naturally. A natural follow up to that is to ask whether we should be doing something to encourage this to happen, like writing more position papers, or just training up more grad students? Should we be trying to actively push for this rather than just relying on people to organically develop an interest in this research area? And I don’t know whether you can actually change research directions in this way, because it’s very far outside my area of expertise, but I’d love someone to study it.

Conversation with Adam Gleave