Superintelligence: Paths, Dangers, Strategies - Nick Bostrom (2014)

Chapter 3. Forms of superintelligence

So what, exactly, do we mean by “superintelligence”? While we do not wish to get bogged down in terminological swamps, something needs to be said to clarify the conceptual ground. This chapter identifies three different forms of superintelligence, and argues that they are, in a practically relevant sense, equivalent. We also show that the potential for intelligence in a machine substrate is vastly greater than in a biological substrate. Machines have a number of fundamental advantages which will give them overwhelming superiority. Biological humans, even if enhanced, will be outclassed.

Many machines and nonhuman animals already perform at superhuman levels in narrow domains. Bats interpret sonar signals better than man, calculators outperform us in arithmetic, and chess programs beat us in chess. The range of specific tasks that can be better performed by software will continue to expand. But although specialized information processing systems will have many uses, there are additional profound issues that arise only with the prospect of machine intellects that have enough general intelligence to substitute for humans across the board.

As previously indicated, we use the term “superintelligence” to refer to intellects that greatly outperform the best current human minds across many very general cognitive domains. This is still quite vague. Different kinds of system with rather disparate performance attributes could qualify as superintelligences under this definition. To advance the analysis, it is helpful to disaggregate this simple notion of superintelligence by distinguishing different bundles of intellectual super-capabilities. There are many ways in which such decomposition could be done. Here we will differentiate between three forms: speed superintelligence, collective superintelligence, and quality superintelligence.

Speed superintelligence

A speed superintelligence is an intellect that is just like a human mind but faster. This is conceptually the easiest form of superintelligence to analyze.¹ We can define speed superintelligence as follows:

Speed superintelligence: A system that can do all that a human intellect can do, but much faster.

By “much” we here mean something like “multiple orders of magnitude.” But rather than try to expunge every remnant of vagueness from the definition, we will entrust the reader with interpreting it sensibly.²

The simplest example of speed superintelligence would be a whole brain emulation running on fast hardware.³ An emulation operating at a speed of ten thousand times that of a biological brain would be able to read a book in a few seconds and write a PhD thesis in an afternoon. With a speedup factor of a million, an emulation could accomplish an entire millennium of intellectual work in one working day.⁴

To such a fast mind, events in the external world appear to unfold in slow motion. Suppose your mind ran at 10,000×. If your fleshly friend should happen to drop his teacup, you could watch the porcelain slowly descend toward the carpet over the course of several hours, like a comet silently gliding through space toward an assignation with a far-off planet; and, as the anticipation of the coming crash tardily propagates through the folds of your friend’s gray matter and from thence out into his peripheral nervous system, you could observe his body gradually assuming the aspect of a frozen oops—enough time for you not only to order a replacement cup but also to read a couple of scientific papers and take a nap.

Because of this apparent time dilation of the material world, a speed superintelligence would prefer to work with digital objects. It could live in virtual reality and deal in the information economy. Alternatively, it could interact with the physical environment by means of nanoscale manipulators, since limbs at such small scales could operate faster than macroscopic appendages. (The characteristic frequency of a system tends to be inversely proportional to its length scale.⁵) A fast mind might commune mainly with other fast minds rather than with bradytelic, molasses-like humans.

The speed of light becomes an increasingly important constraint as minds get faster, since faster minds face greater opportunity costs in the use of their time for traveling or communicating over long distances.⁶ Light is roughly a million times faster than a jet plane, so it would take a digital agent with a mental speedup of 1,000,000× about the same amount of subjective time to travel across the globe as it does a contemporary human journeyer. Dialing somebody long distance would take as long as getting there “in person,” though it would be cheaper as a call would require less bandwidth. Agents with large mental speedups who want to converse extensively might find it advantageous to move near one another. Extremely fast minds with need for frequent interaction (such as members of a work team) may take up residence in computers located in the same building to avoid frustrating latencies.

Collective superintelligence

Another form of superintelligence is a system achieving superior performance by aggregating large numbers of smaller intelligences:

Collective superintelligence: A system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system.

Collective superintelligence is less conceptually clear-cut than speed superintelligence.⁷ However, it is more familiar empirically. While we have no experience with human-level minds that differ significantly in clock speed, we dohave ample experience with collective intelligence, systems composed of various numbers of human-level components working together with various degrees of efficiency. Firms, work teams, gossip networks, advocacy groups, academic communities, countries, even humankind as a whole, can—if we adopt a somewhat abstract perspective—be viewed as loosely defined “systems” capable of solving classes of intellectual problems. From experience, we have some sense of how easily different tasks succumb to the efforts of organizations of various size and composition.

Collective intelligence excels at solving problems that can be readily broken into parts such that solutions to sub-problems can be pursued in parallel and verified independently. Tasks like building a space shuttle or operating a hamburger franchise offer myriad opportunities for division of labor: different engineers work on different components of the spacecraft; different staffs operate different restaurants. In academia, the rigid division of researchers, students, journals, grants, and prizes into separate self-contained disciplines—though unconducive to the type of work represented by this book—might (only in a conciliatory and mellow frame of mind) be viewed as a necessary accommodation to the practicalities of allowing large numbers of diversely motivated individuals and teams to contribute to the growth of human knowledge while working relatively independently, each plowing their own furrow.

A system’s collective intelligence could be enhanced by expanding the number or the quality of its constituent intellects, or by improving the quality of their organization.⁸ To obtain a collective superintelligence from any present-day collective intelligence would require a very great degree of enhancement. The resulting system would need to be capable of vastly outperforming any current collective intelligence or other cognitive system across many very general domains. A new conference format that lets scholars exchange information more effectively, or a new collaborative information-filtering algorithm that better predicted users’ ratings of books and movies, would clearly not on its own amount to anything approaching collective superintelligence. Nor would a 50% increase in the world population, or an improvement in pedagogical method that enabled students to complete a school day in four hours instead of six. Some far more extreme growth of humanity’s collective cognitive capacity would be required to meet the criterion of collective superintelligence.

Note that the threshold for collective superintelligence is indexed to the performance levels of the present—that is, the early twenty-first century. Over the course of human prehistory, and again over the course of human history, humanity’s collective intelligence has grown by very large factors. World population, for example, has increased by at least a factor of a thousand since the Pleistocene.⁹ On this basis alone, current levels of human collective intelligence could be regarded as approaching superintelligence relative to a Pleistocene baseline. Some improvements in communications technologies—especially spoken language, but perhaps also cities, writing, and printing—could also be argued to have, individually or in combination, provided super-sized boosts, in the sense that if another innovation of comparable impact to our collective intellectual problem-solving capacity were to happen, it would result in collective superintelligence.¹⁰

A certain kind of reader will be tempted at this point to interject that modern society does not seem so particularly intelligent. Perhaps some unwelcome political decision has just been made in the reader’s home country, and the apparent unwisdom of that decision now looms large in the reader’s mind as evidence of the mental incapacity of the modern era. And is it not the case that contemporary humanity is idolizing material consumption, depleting natural resources, polluting the environment, decimating species diversity, all the while failing to remedy screaming global injustices and neglecting paramount humanistic or spiritual values? However, setting aside the question of how modernity’s shortcomings stack up against the not-so-inconsiderable failings of earlier epochs, nothing in our definition of collective superintelligence implies that a society with greater collective intelligence is necessarily better off. The definition does not even imply that the more collectively intelligent society is wiser. We can think of wisdom as the ability to get the important things approximately right. It is then possible to imagine an organization composed of a very large cadre of very efficiently coordinated knowledge workers, who collectively can solve intellectual problems across many very general domains. This organization, let us suppose, can operate most kinds of businesses, invent most kinds of technologies, and optimize most kinds of processes. Even so, it might get a few key big-picture issues entirely wrong—for instance, it may fail to take proper precautions against existential risks—and as a result pursue a short explosive growth spurt that ends ingloriously in total collapse. Such an organization could have a very high degree of collective intelligence; if sufficiently high, the organization is a collective superintelligence. We should resist the temptation to roll every normatively desirable attribute into one giant amorphous concept of mental functioning, as though one could never find one admirable trait without all the others being equally present. Instead, we should recognize that there can exist instrumentally powerful information processing systems—intelligent systems—that are neither inherently good nor reliably wise. But we will revisit this issue in Chapter 7.

Collective superintelligence could be either loosely or tightly integrated. To illustrate a case of loosely integrated collective superintelligence, imagine a planet, MegaEarth, which has the same level of communication and coordination technologies that we currently have on the real Earth but with a population one million times as large. With such a huge population, the total intellectual workforce on MegaEarth would be correspondingly larger than on our planet. Suppose that a scientific genius of the caliber of a Newton or an Einstein arises at least once for every 10 billion people: then on MegaEarth there would be 700,000 such geniuses living contemporaneously, alongside proportionally vast multitudes of slightly lesser talents. New ideas and technologies would be developed at a furious pace, and global civilization on MegaEarth would constitute a loosely integrated collective superintelligence.¹¹

If we gradually increase the level of integration of a collective intelligence, it may eventually become a unified intellect—a single large “mind” as opposed to a mere assemblage of loosely interacting smaller human minds.¹² The inhabitants of MegaEarth could take steps in that direction by improving communications and coordination technologies and by developing better ways for many individuals to work on any hard intellectual problem together. A collective superintelligence could thus, after gaining sufficiently in integration, become a “quality superintelligence.”

Quality superintelligence

We can distinguish a third form of superintelligence.

Quality superintelligence: A system that is at least as fast as a human mind and vastly qualitatively smarter.

As with collective intelligence, intelligence quality is also a somewhat murky concept; and in this case the difficulty is compounded by our lack of experience with any variations in intelligence quality above the upper end of the present human distribution. We can, however, get some grasp of the notion by considering some related cases.

First, we can expand the range of our reference points by considering nonhuman animals, which have intelligence of lower quality. (This is not meant as a speciesist remark. A zebrafish has a quality of intelligence that is excellently adapted to its ecological needs; but the relevant perspective here is a more anthropocentric one: our concern is with performance on humanly relevant complex cognitive tasks.) Nonhuman animals lack complex structured language; they are capable of no or only rudimentary tool use and tool construction; they are severely restricted in their ability to make long-term plans; and they have very limited abstract reasoning ability. Nor are these limitations fully explained by a lack of speed or of collective intelligence among nonhuman animal minds. In terms of raw computational power, human brains are probably inferior to those of some large animals, including elephants and whales. And although humanity’s complex technological civilization would be impossible without our massive advantage in collective intelligence, not all distinctly human cognitive capabilities depend on collective intelligence. Many are highly developed even in small, isolated hunter-gatherer bands.¹³ And many are not nearly as highly developed among highly organized nonhuman animals, such as chimpanzees and dolphins intensely trained by human instructors, or ants living in their own large and well-ordered societies. Evidently, the remarkable intellectual achievements of Homo sapiens are to a significant extent attributable to specific features of our brain architecture, features that depend on a unique genetic endowment not shared by other animals. This observation can help us illustrate the concept of quality superintelligence: it is intelligence of quality at least as superior to that of human intelligence as the quality of human intelligence is superior to that of elephants’, dolphins’, or chimpanzees’.

A second way to illustrate the concept of quality superintelligence is by noting the domain-specific cognitive deficits that can afflict individual humans, particularly deficits that are not caused by general dementia or other conditions associated with wholesale destruction of the brain’s neurocomputational resources. Consider, for example, individuals with autism spectrum disorders who may have striking deficits in social cognition while functioning well in other cognitive domains; or individuals with congenital amusia, who are unable to hum or recognize simple tunes yet perform normally in most other respects. Many other examples could be adduced from the neuropsychiatric literature, which is replete with case studies of patients suffering narrowly circumscribed deficits caused by genetic abnormalities or brain trauma. Such examples show that normal human adults have a range of remarkable cognitive talents that are not simply a function of possessing a sufficient amount of general neural processing power or even a sufficient amount of general intelligence: specialized neural circuitry is also needed. This observation suggests the idea of possible but non-realized cognitive talents, talents that no actual human possesses even though other intelligent systems—ones with no more computing power than the human brain—that did have those talents would gain enormously in their ability to accomplish a wide range of strategically relevant tasks.

Accordingly, by considering nonhuman animals and human individuals with domain-specific cognitive deficits, we can form some notion of different qualities of intelligence and the practical difference they make. Had Homo sapiens lacked (for instance) the cognitive modules that enable complex linguistic representations, it might have been just another simian species living in harmony with nature. Conversely, were we to gain some new set of modules giving an advantage comparable to that of being able to form complex linguistic representations, we would become superintelligent.

Direct and indirect reach

Superintelligence in any of these forms could, over time, develop the technology necessary to create any of the others. The indirect reaches of these three forms of superintelligence are therefore equal. In that sense, the indirect reach of current human intelligence is also in the same equivalence class, under the supposition that we are able eventually to create some form of superintelligence. Yet there is a sense in which the three forms of superintelligence are much closer to one another: any one of them could create other forms of superintelligence more rapidly than we can create any form of superintelligence from our present starting point.

The direct reaches of the three different forms of superintelligence are harder to compare. There may be no definite ordering. Their respective capabilities depend on the degree to which they instantiate their respective advantages—how fast a speed superintelligence is, how qualitatively superior a quality superintelligence is, and so forth. At most, we might say that, ceteris paribus, speed superintelligence excels at tasks requiring the rapid execution of a long series of steps that must be performed sequentially while collective superintelligence excels at tasks admitting of analytic decomposition into parallelizable sub-tasks and tasks demanding the combination of many different perspectives and skill sets. In some vague sense, quality superintelligence would be the most capable form of all, inasmuch as it could grasp and solve problems that are, for all practical purposes, beyond the direct reach of speed superintelligence and collective superintelligence.¹⁴

In some domains, quantity is a poor substitute for quality. One solitary genius working out of a cork-lined bedroom can write In Search of Lost Time. Could an equivalent masterpiece be produced by recruiting an office building full of literary hacks?¹⁵ Even within the range of present human variation we see that some functions benefit greatly from the labor of one brilliant mastermind as opposed to the joint efforts of myriad mediocrities. If we widen our purview to include superintelligent minds, we must countenance a likelihood of there being intellectual problems solvable only by superintelligence and intractable to any ever-so-large collective of non-augmented humans.

There might thus be some problems that are solvable by a quality superintelligence, and perhaps by a speed superintelligence, yet which a loosely integrated collective superintelligence cannot solve (other than by first amplifying its own intelligence).¹⁶ We cannot clearly see what all these problems are, but we can characterize them in general terms.¹⁷ They would tend to be problems involving multiple complex interdependencies that do not permit of independently verifiable solution steps: problems that therefore cannot be solved in a piecemeal fashion, and that might require qualitatively new kinds of understanding or new representational frameworks that are too deep or too complicated for the current edition of mortals to discover or use effectively. Some types of artistic creation and strategic cognition might fall into this category. Some types of scientific breakthrough, perhaps, likewise. And one can speculate that the tardiness and wobbliness of humanity’s progress on many of the “eternal problems” of philosophy are due to the unsuitability of the human cortex for philosophical work. On this view, our most celebrated philosophers are like dogs walking on their hind legs—just barely attaining the threshold level of performance required for engaging in the activity at all.¹⁸

Sources of advantage for digital intelligence

Minor changes in brain volume and wiring can have major consequences, as we see when we compare the intellectual and technological achievements of humans with those of other apes. The far greater changes in computing resources and architecture that machine intelligence will enable will probably have consequences that are even more profound. It is difficult, perhaps impossible, for us to form an intuitive sense of the aptitudes of a superintelligence; but we can at least get an inkling of the space of possibilities by looking at some of the advantages open to digital minds. The hardware advantages are easiest to appreciate:

✵ Speed of computational elements. Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~ 2 GHz).¹⁹ As a consequence, the human brain is forced to rely on massive parallelization and is incapable of rapidly performing any computation that requires a large number of sequential operations.²⁰ (Anything the brain does in under a second cannot use much more than a hundred sequential operations—perhaps only a few dozen.) Yet many of the most practically important algorithms in programming and computer science are not easily parallelizable. Many cognitive tasks could be performed far more efficiently if the brain’s native support for parallelizable pattern-matching algorithms were complemented by, and integrated with, support for fast sequential processing.

✵ Internal communication speed. Axons carry action potentials at speeds of 120 m/s or less, whereas electronic processing cores can communicate optically at the speed of light (300,000,000 m/s).²¹ The sluggishness of neural signals limits how big a biological brain can be while functioning as a single processing unit. For example, to achieve a round-trip latency of less than 10 ms between any two elements in a system, biological brains must be smaller than 0.11 m³. An electronic system, on the other hand, could be 6.1×10¹⁷ m³, about the size of a dwarf planet: eighteen orders of magnitude larger.²²

✵ Number of computational elements. The human brain has somewhat fewer than 100 billion neurons.²³ Humans have about three and a half times the brain size of chimpanzees (though only one-fifth the brain size of sperm whales).²⁴The number of neurons in a biological creature is most obviously limited by cranial volume and metabolic constraints, but other factors may also be significant for larger brains (such as cooling, development time, and signal-conductance delays—see the previous point). By contrast, computer hardware is indefinitely scalable up to very high physical limits.²⁵ Supercomputers can be warehouse-sized or larger, with additional remote capacity added via high-speed cables.²⁶

✵ Storage capacity. Human working memory is able to hold no more than some four or five chunks of information at any given time.²⁷ While it would be misleading to compare the size of human working memory directly with the amount of RAM in a digital computer, it is clear that the hardware advantages of digital intelligences will make it possible for them to have larger working memories. This might enable such minds to intuitively grasp complex relationships that humans can only fumblingly handle via plodding calculation.²⁸ Human long-term memory is also limited, though it is unclear whether we manage to exhaust its storage capacity during the course of an ordinary lifetime—the rate at which we accumulate information is so slow. (On one estimate, the adult human brain stores about one billion bits—a couple of orders of magnitude less than a low-end smartphone.²⁹) Both the amount of information stored and the speed with which it can be accessed could thus be vastly greater in a machine brain than in a biological brain.

✵ Reliability, lifespan, sensors, etc. Machine intelligences might have various other hardware advantages. For example, biological neurons are less reliable than transistors.³⁰ Since noisy computing necessitates redundant encoding schemes that use multiple elements to encode a single bit of information, a digital brain might derive some efficiency gains from the use of reliable high-precision computing elements. Brains become fatigued after a few hours of work and start to permanently decay after a few decades of subjective time; microprocessors are not subject to these limitations. Data flow into a machine intelligence could be increased by adding millions of sensors. Depending on the technology used, a machine might have reconfigurable hardware that can be optimized for changing task requirements, whereas much of the brain’s architecture is fixed from birth or only slowly changeable (though the details of synaptic connectivity can change over shorter timescales, like days).³¹

At present, the computational power of the biological brain still compares favorably with that of digital computers, though top-of-the-line supercomputers are attaining levels of performance that are within the range of plausible estimates of the brain’s processing power.³² But hardware is rapidly improving, and the ultimate limits of hardware performance are vastly higher than those of biological computing substrates.

Digital minds will also benefit from major advantages in software:

✵ Editability. It is easier to experiment with parameter variations in software than in neural wetware. For example, with a whole brain emulation one could easily trial what happens if one adds more neurons in a particular cortical area or if one increases or decreases their excitability. Running such experiments in living biological brains would be far more difficult.

✵ Duplicability. With software, one can quickly make arbitrarily many high-fidelity copies to fill the available hardware base. Biological brains, by contrast, can be reproduced only very slowly; and each new instance starts out in a helpless state, remembering nothing of what its parents learned in their lifetimes.

✵ Goal coordination. Human collectives are replete with inefficiencies arising from the fact that it is nearly impossible to achieve complete uniformity of purpose among the members of a large group—at least until it becomes feasible to induce docility on a large scale by means of drugs or genetic selection. A “copy clan” (a group of identical or almost identical programs sharing a common goal) would avoid such coordination problems.

✵ Memory sharing. Biological brains need extended periods of training and mentorship whereas digital minds could acquire new memories and skills by swapping data files. A population of a billion copies of an AI program could synchronize their databases periodically, so that all the instances of the program know everything that any instance learned during the previous hour. (Direct memory transfer requires standardized representational formats. Easy swapping of high-level cognitive content would therefore not be possible between just any pair of machine intelligences. In particular, it would not be possible among first-generation whole brain emulations.)

✵ New modules, modalities, and algorithms. Visual perception seems to us easy and effortless, quite unlike solving textbook geometry problems—this despite the fact that it takes a massive amount of computation to reconstruct, from the two-dimensional patterns of stimulation on our retinas, a three-dimensional representation of a world populated with recognizable objects. The reason this seems easy is that we have dedicated low-level neural machinery for processing visual information. This low-level processing occurs unconsciously and automatically, without draining our mental energy or conscious attention. Music perception, language use, social cognition, and other forms of information processing that are “natural” for us humans seem to be likewise supported by dedicated neurocomputational modules. An artificial mind that had such specialized support for other cognitive domains that have become important in the contemporary world—such as engineering, computer programming, and business strategy—would have big advantages over minds like ours that have to rely on clunky general-purpose cognition to think about such things. New algorithms may also be developed to take advantage of the distinct affordances of digital hardware, such as its support for fast serial processing.

The ultimately attainable advantages of machine intelligence, hardware and software combined, are enormous.³³ But how rapidly could those potential advantages be realized? That is the question to which we now turn.