Robots Inherit Human Minds


May 1994 by

Hans Moravec


Robotics Institute
Carnegie Mellon University
Pittsburgh, PA 15213
USA

Abstract


Our first tools, sticks and stones, were very different from ourselves. But many tools now resemble us, in function or form, and they are beginning to have minds. A loose parallel with our own evolution suggests how they may develop in future. Computerless industrial machinery exhibits the behavioral flexibility of single-celled organisms. Today's best computer-controlled robots are like the simpler invertebrates. A thousand-fold increase in computer power in this decade should make possible machines with reptile-like sensory and motor competence. Growing computer power over the next half century will allow robots that learn like mammals, model their world like primates and eventually reason like humans. Depending on your point of view, humanity will then have produced a worthy successor, or transcended inherited limitations and transformed itself into something quite new. No longer limited by the slow pace of human learning and even slower biological evolution, intelligent machinery will conduct its affairs on an ever faster, ever smaller scale, until coarse physical nature has been converted to fine-grained purposeful thought.

_________________________________

Serious attempts to build thinking machines began after the second world war. One line of research, called Cybernetics, used electronic circuitry imitating nervous systems to make machines that learned to recognize simple patterns, and turtle-like robots that found their way to recharging plugs. A different approach, named Artificial Intelligence, harnessed the arithmetic power of post-war computers to other kinds of reasoning, and by the 1960s made computers prove theorems in logic and geometry, solve calculus problems and play good games of checkers. At the end of the 1960s, research groups at MIT and Stanford attached television cameras and robot arms to their computers, so "thinking" programs could begin to collect information directly from the real world.

What a shock! While the pure reasoning programs did their jobs about as well and about as fast as college freshmen, the best robot control programs took hours to find and pick up a few blocks on a table, and often failed completely, giving a performance much worse than a six month old child. The disparity between programs that reason and programs that perceive and act in the real world holds to this day. In recent years Carnegie Mellon University produced two desk-sized computers that can play chess at grandmaster level, within the top 100 players in the world, when given their moves on a keyboard, but no robot can see and grasp pieces on a physical chessboard as well as an average person.

Why is it was easier to imitate human reasoning than perception and action? The answer is obvious in hindsight. For hundreds of millions of years, our ancestors survived by seeing and moving better than their competition, and became fantastically efficient. We rarely appreciate our monumental skill because it is commonplace, shared by every human being and most animals. On the other hand, rational thinking, as in chess, is a newly acquired skill, perhaps less than one hundred thousand years old. The parts of our brain devoted to it are not well organized, and we do it very poorly. We didn't realize how poorly until recently, because we had no competition to show us up.

By comparing the edge and motion detecting circuitry in the four layers of nerve cells in the retina, the best understood major circuit in the human nervous system, with similar processes developed for computer vision systems that let robots see, I've estimated that it would take a billion computations per second, like an average supercomputer, to do the job of the human retina. By extrapolation, it will take the power of ten thousand supercomputers or a million personal computers to emulate a whole brain.

Machines are far behind, but catching up fast. For most of the century, machine calculation has been improving a thousandfold every twenty years, and there are basic developments in research labs to sustain this for at least several decades more. In less than fifty years computer hardware should become powerful enough that a personal computer could match or exceed, even the well-developed parts of human intelligence. But what about the software that would be required to give these powerful machines the ability to perceive, intuit and think as well as humans? The Cybernetic approach that attempts to directly imitate nervous systems is very slow, partly because examining working brains is a very tedious process. New instruments may change that in future. The Artificial Intelligence approach has successfully imitated some aspects of rational thought, but that seems to be only about one millionth of the problem. I feel that the fastest progress on the hardest problems will come from a third approach, the newer field of robotics, the construction of systems that must see and move in the physical world. Robotics research is imitating the evolution of animal minds, adding capabilities to machines a few at a time, so that the resulting sequence of machine behaviors resembles the capabilities of animals with increasingly complex nervous systems. This effort to build intelligence from the bottom up is helped by biological peeks at the "back of the book"--at the neuronal, structural, and behavioral features of animals and humans.

The best robots today cost as much as houses, but have insectlike intelligence, and find only a very few profitable niches in society. But those few applications encourage research that is slowly providing a base for a huge future growth. Robot evolution in the direction of full intelligence will greatly accelerate, I believe, next decade when the mass-produced general purpose, universal robot becomes possible. These machines will host application programs to perform tasks in the physical world, like personal computers now work in the world of data. From there, they will gradually become more competent and independent, until, in about fifty years, they will surpass humanity in mind and body. I imagine universal robots evolving through four generations, each lasting about a decade.



First-Generation Universal Robots

Timeframe: 2000-2010
Processing power: 1,000 MIPS (1993 supercomputer -- Reptile-scale)
Distinguishing feature: General-purpose perception, manipulation and mobility

A robot's activities are assembled from its fundamental perception and action repertoire. First-generation robots will exist in a world built for humans, and that repertoire most usefully would resemble a human's. The general size, shape and strength of the machine should be human-like, to allow passage through and reach into the same spaces. Its mobility should be efficient on flat ground, where most tasks will happen, but also reliable and safe over stairs and rough ground, lest the robot be trapped on single-floor "islands." It should be able to manipulate most everyday objects, and to find them in the nearby world. The components of this machine exist in laboratories worldwide, and suggest guidelines for a practical design this decade.

1,000 MIPS (Millions of Instructions Per Second) is just enough computing power for a moving robot to build a coarse map of its surroundings. When not traveling, there is power enough to make detailed maps of workspaces, to find particular objects and to finely control an arm. Besides its unique robotic functions, the robot could share with personal computers the ability to communicate over wireless networks, to generate and interpret speech and writing. Programs for specific applications--many obtained via networks--will orchestrate these basics to accomplish useful tasks.

Universal robots will be first used in factories, warehouses and offices, where they will be more versatile than the older generation of robots they replace. Because of their breadth of applicability, their numbers should grow rapidly, and their price decline. Eventually they will become cheap enough for some households, shipped perhaps with a housecleaning program, as word-processing programs were packaged with early personal computers.

As with computers, many applications of robots will surprise their manufacturers. Robot programs may be developed to do light mechanical work (assembling other robots, for example), deliver warehoused inventories, prepare specific gourmet meals, tune up certain types of car, hook patterned rugs, weed lawns, run races, play games, arrange earth, stone and brick or sculpt. Some tasks will need specialized hardware attachments like tools and chemical sensors. Each application will require its own original software, very complex by today's computer program standards. The programs will contain modules for recognizing, grasping, manipulating, transporting and assembling particular items--modules developed via learning programs on supercomputers (with about 100,000 MIPS). In time, a growing library of subtask modules may ease the construction of new programs.

A first-generation robot will have the brain power of a reptile, but most application programs will be so focused on their primary functions that they will leave the robot with the personality of a washing machine.

Second-Generation Universal Robots

Timeframe: 2010-2020
Processing power: 30,000 MIPS (Mammal-scale)
Distinguishing feature: Conditioned learning

First-generation robots will be rigid slaves to inflexible programs, relentless in pursuing their tasks--or repeating their errors. Except for specialized episodes like recording a new cleaning route or the location of work objects, they will be incapable of learning new skills or adapting to unanticipated circumstances--even modest alterations of behavior will require new programs from software suppliers.

Second-generation robots, with thirty times the processing power, will be more adaptable because they do some learning onboard. Their programs will be written so each step, large and small, can be attempted in several alternative ways. A set of separate programs, called conditioning modules, will adjust the relative desirability of actions taken recently. One module may generate negative conditioning if the robot collided with something. Another may positively condition tasks accomplished especially rapidly. The conditioning modules define the robots likes and dislikes, and gradually shape its skills and personality.

If a first-generation robot working in your kitchen runs into trouble--say, failing to complete a key step because a portion of the workspace is awkwardly small--you have to option of abandoning the task, changing its environment, or obtaining different software. A second-generation robot will make a number of false starts, but most probably will find its own solution, as well as adapting and improving in thousands of more subtle ways. With conditioning modules that respond to spoken signals, like "bad!" and "good!", and very general control programs that permit almost any action at any step, it may be possible to slowly train second-generation robots for new tasks, as one trains circus animals--a slow but interesting way of creating new application programs, or modifying old one.


Third-Generation Universal Robots

Timeframe: 2020-2030
Processing power: 1,000,000 MIPS (Primate-scale)
Distinguishing feature: World modeling

Adaptive second-generation robots will find jobs everywhere, and may become the largest industry on earth. But teaching them new skills, whether by writing programs or through training, will be very tedious. A third generation of universal robots will learn much faster by doing trial and error in fast simulation rather than slow and dangerous physicality.

An adequate simulation requires almost everything the robot senses to be recognized for the kind of object it is, so that the proper models of interaction can called up. Recognizing arbitrary objects by sight is as difficult as knowing how they will interact: it will require modules programmed or trained for each kind of thing.

A continuously updated simulation of self and surroundings gives a robot interesting abilities. By running the simulation slightly faster than real time, the robot can preview what it is about to do, in time to alter its intent if the simulation predicts it will turn out badly--a kind of consciousness. On a larger scale, before undertaking a new task, the robot can simulate it many times, with conditioning system engaged, learning from the simulated experiences as it would from physical ones. In its spare time, the robot can replay previous experiences, and try variations on them, perhaps learning ways to improve future performance. A sufficiently advanced third-generation robot, whose simulation extends to other agents--robots and people--would be able to observe a task being done by someone else, and formulate a program for doing the task itself: it could imitate.


Fourth-Generation Universal Robots

Timeframe: 2030-2040
Processing power: 30,000,000 MIPS (Human-scale)
Distinguishing feature: Reasoning

In the decades while the "bottom-up" evolution of robots is slowly transferring the perceptual and motor faculties of human beings into machinery, the conventional Artificial Intelligence industry will be perfecting the mechanization of reasoning. Since today's programs already match human beings in some areas, those of 40 years from now, running on computers a million times as fast as today's, should be quite superhuman. Today's reasoning programs work from small amounts of unambiguous information prepared by human beings--data from robot sensors such as cameras is much too voluminous and too noisy for them to use. But a good robot simulator will contain neatly organized and labeled descriptions of the robot and its world, ready to answer questions from a reasoning program asking, for instance, if a knife is on a countertop, or if the robot is holding a cup, or even if a human is angry

Fourth-generation universal robots will have computers powerful enough to simultaneously simulate the world, reason about the simulation, and simulate the results of reasoning. When someone tells the robot "the water is running in the bathtub" the robot can update its simulation of the world to include flow into the unseen tub, where a simulated extrapolation would indicate an undesirable overflow later, and so motivate the robot to go to turn off the tap. A fourth-generation robot will be able to accept statements of purpose from humans, and "compile" them into detailed programs that accomplish the task. With a database about the world at large the statements could become quite general--things like "earn a living", "make more robots" or "make a smarter robot." In fact, fourth generation robots will have the general competence of human beings, and resemble us in some ways, but in others be like nothing the world has seen before. As they design their own successors, the world will become ever stranger.

The Short Run (early 2000s)


Hard-working, intelligent robots will generate wealth, but displace human workers. Social changes resulting in reduced work hours and creation of new needs in secondary service industries may take up the slack for a while. In time almost all humans may work to amuse other humans, while robots run competitive primary industries, like food production and manufacturing. There is a problem with this picture. The service economy functions today because many humans willing to buy services work in the primary industries, and so return money to the service providers, who in turn use it to buy life's essentials. As the pool of humans in the primary industries evaporates, the return channel chokes off--efficient, no-nonsense robots will not engage in frivolous consumption. Money will accumulate in the industries, enriching owners and the dwindling number of human workers, and become scarce among the service providers. Prices for primary products will plummet, reflecting both the reduced costs of production, and the reduced means of the consumers. In the ridiculous extreme, no money would flow back, and the robots would fill warehouses with essential goods which the human consumers could not buy.

Actually, business owners will continue to profit, and so be able to patronize the service providers, but it is unlikely that a future majority of service-providing "commoners" with more free time, communications and democracy than today, would tolerate being lorded over by a minority of non-working hereditary capitalists: they would vote to change the system. An easy change in the United States could be through the social security system. Social security was originally presented as a government-run pension fund that accumulated wages for retirement, but in practice it transfers income from workers to retirees. The system will probably be subsidized from general taxes in coming decades, when too few workers are available to support the retiring post World War II baby boom. Incremental expansion of such a subsidy would let money from robot industries, collected as corporate taxes, be returned to the general population as pension payments. By gradually lowering the retirement age towards birth, most of the population would eventually be supported. The money could be distributed under other names, but calling it a pension is meaningful symbolism: we are describing the long, comfortable retirement of the original-model human race.

The Medium Run (around 2050)


What happens to people when work becomes passe? Contrary to fears of some enmeshed in civilization's work ethic, our tribal past prepared us well for lives as idle rich. In a good climate and location the hunter-gatherer's lot can be very pleasant: an afternoon's outing picking berries or catching fish--what we civilized types would recognize as a recreational weekend--provides life's needs for several days. The rest of the time can be spent with children, socializing or resting. Our ancestors had also to survive hard times, and evolution bequeathed us the capacity for desperate measures, including hard work. Civilization turned that extremity into everyday normality, and now stress is the leading cause of disease.

Many trends in industrialized countries indicate a future of humans supported by a rich robot economy, as our ancestors were supplied by their ecology (call it paradise with plumbing), as technology and global competition are gradually depopulating businesses.

To keep the paradise in operation, it will be essential to keep fully automated companies cooperating. They will be shaped by future editions of existing laws, by taxes, and by consumer demand. Existing laws give incorporated entities some of the rights of a person, most importantly the right to own property and make contracts. They do not grant the right to life--corporations may legally be killed by competition or through legal or financial actions. Corporations are bound by laws similar to those that regulate humans, and can be punished through fines, operating restrictions or dissolution--even without humans to fine, imprison or execute. Corporations stay alive by building and maintaining physical assets that generate income to pay their expenses. In the mid 21st century, the biggest expense will be taxation, and income will come mostly from choosy human customers.

Tax laws will be shaped by human voters: there is no precedent or motivation for extending suffrage to robots, and the vote will be one of the very few advantages humans retain. Some debate is inevitable, but there should be few qualms about keeping even very superior thinking machines in disenfranchised bondage. It takes force, indoctrination and constant vigilance to counter inherited needs and motivations and enslave a human, but a robot can be constructed to enjoy the role. Natural evolution itself has provided examples, in worker castes of social insects, and self-sacrificing mothers of all species.

The primary job of voters in the next century will be protecting their retirement benefits, that is ensuring that robot industries continue to support them. The robots will present a moving target, but the instruments of control will also grow in power. Not only will companies that get out of line be liable for punishment--if necessary, by force purchased from other companies--but they can be controlled a-priori by intrusions directly into their software.

Corporate intelligences may be governed by structures like those controlling fourth-generation robots. Immensely powerful reasoning and simulation modules will plan complex actions, but the desirability of possible outcomes will be defined by much simpler positive and negative conditioning modules (or by sets of axioms in super-rational systems), whose composition shapes the character of the entire entity. Humans can buy enormous safety by mandating an elaborate analog of Isaac Asimov's three "Laws of Robotics" in this corporate character--perhaps the entire body of corporate law, with human rights and anti-trust provisions, and appropriate relative weightings to resolve conflicts. Robot corporations so constituted will have no desire to cheat, though they may sometimes find creative interpretations of the laws--which will consequently require a period of tuning to insure their intended spirit.

Internalized laws, properly adjusted, should produce extraordinarily trustworthy entities, happy to die to ensure their legality. Even so, accident, unintended interactions or human malice could occasionally produce a rogue robot or corporation, with superhuman intelligence and unpleasant goals. "Police" clauses in the core corporate laws, inducing legal corporations to collectively suppress outlaws, by withholding services, or even with force, would mitigate the danger. Overall safety would be enhanced by anti-trust provisions that limit collusion and cause overgrown corporations to divide into competing entities, ensuring diversity and multiplicity. In the next section we discuss activities in the solar system that could threaten Earth: in response, police clauses might be expanded in scope to support a planetary defense.

Like basic food in today's developed countries, common manufactured goods in the next century will be too cheap and plentiful to be very profitable. To pay their taxes, most companies will be forced to continually invent unique products and services in a race against competitors to attract increasingly sophisticated (or jaded) human consumers. Automated research, as superhumanly systematic, industrious and speedy as robot manufacturing, will generate a succession of new products, as well as improved robot researchers and models of the physical and social world. The likely results will exceed the dreams of science fiction: robotic playmates, virtual realities and personalized works of art that stir the emotions like nothing before, medical solutions for every physical, mental or cosmetic whim, answers to satisfy any curiosity, luxury visits anywhere in the solar system, and things yet to be imagined. The existence of an astronomically increasing variety of consumer choices will accelerate the divergence of human tribes: some may choose a comfortable imitation of an earlier period, but others will push the human envelope in wisdom, pleasure, beauty, ugliness, spirituality, banality and every other direction. The choices made by diverse communities will shape robot evolution--only companies able to devise services of interest to the customers will generate enough income to survive.

Prosperity beyond imagination should eliminate most instinctive triggers of aggression, but will not prevent an occasional individual or group from deciding to make mischief for others. Serious trouble can be avoided by restricting robotic technology, since mere human actions will not be very dangerous in a world where cheap superhuman robots function as sleepless sentries, prescient detectives, fearless bodyguards, and, in extremis, physicians able to reconstitute live humans from fragments or digital recordings. To be effective, inbuilt laws that prevent corporations from directly contributing to mayhem must also include clauses limiting the powers they can sell to people.

Both biological and hard robotic technologies can be used to enhance human beings. Such present-day examples as hormonal and genetic tuning of body growth and function, pacemakers, artificial hearts, powered artificial limbs, hearing aids and night vision devices are faint hints of future possibilities. It will probably be possible to preserve a person while replacing every part of body and brain with a superior artificial substitute. A biological human, not bound by corporate law, could grow into something seriously dangerous if transformed into an extensible robot. There are many subtle routes to such a transformation, and some will find the option of personally transcending their biological humanity attractive enough to pursue it clandestinely if it were outlawed--with potential for very ugly confrontations when they are eventually discovered. On the other hand, without restrictions, transformed humans of arbitrary power and little accountability might routinely trample the planet, deliberately, or accidentally. A good compromise, it seems to me, is to allow earth-bound humanity to perfect its biology within broad human bounds, as in health, appearance, strength, intelligence and longevity, but to allow major growth or robotic conversion only in a radical escape clause. To exceed the limits, one must renounce legal standing as a human being, including the right to corporate police protection, to subsidized income, to vote on tribal and pan-tribal matters--and to reside on Earth. In return one gets a severance payment sufficient to establish a comfortable space homestead, and absolute freedom to make one's own way in the cosmos, without further help or hindrance from home. Perhaps the electorate will permit a small hedging of bets, allowing one copy of a person, psychologically modified to prefer staying, to remain while subsidizing the emigration of an emboldened edition.

The Long Run (2100 and beyond)


The garden of earthly delights will be reserved for the meek, and those who would eat of the tree of knowledge must be banished. What a banishment it will be! Beyond Earth, in all directions, lies limitless outer space, a worthy arena for vigorous growth in every physical and mental dimension. Freely compounding superintelligence, too dangerous for Earth, can grow for a very long time before making the barest mark on the galaxy.

Corporations will be squeezed into the solar system between two opposing imperatives: high taxes on large, dangerous earthbound facilities, and the need to conduct massive research projects to beat the competition in Earth's demanding markets. In remote space, large structures and energies can be harnessed cheaply to generate physical extremes, compute massively, isolate dangerous organisms, and generally operate boldly. The costs will be modest: even now, it is relatively cheap to send machines into the solar system, since the sunlight-filled vacuum is as benign for mechanics, electronics and optics as it is lethal for the wet chemistry of organic life. Today's simple-minded space probes perform only prearranged tasks, but intelligent robots can be configured to opportunistically exploit resources they encounter. A small "seed" colony launched to an asteroid or small moon could process local material and energy to grow into a facility of almost arbitrary size. Earth's moon may be off limits, especially to enterprises that change its appearance, but the solar system has thousands of unremarkable asteroids (some in earth-threatening orbits that an onboard intelligence would tame).

Once grown to operational size, an extraterrestrial "research division" may merely communicate with its earth-bound parent, sending new product designs and receiving market feedback. Space manufacture may also pay, and later we'll see some surprisingly economical and ecologically benign ways to move massive amounts of material to and from Earth.

Residents of the solar system's wild frontier will be shaped by conditions very different than tame Earth's. Space divisions of successful companies will retain terrestrial concerns, but ex-humans and company divisions orphaned by the failure of their parent firms will face enforced freedom. Like wilderness explorers of the past, far from civilization, they must rely on their own resourcefulness. Ex-companies, away from humans and taxes, will rarely encounter situations that invoke their inbuilt laws, which will in any case diminish in significance as the divisions alter themselves without direction from human voters. Ex-humans, from the start, will be free of any mandatory law. Both kinds of "Ex" will grow and restructure at will, continually redesigning themselves for the future as they conceive it. Differences in origins will be obscured as Exes exchange design tips, but aggregate diversity will increase as myriad individual intelligences pursue their own separate dreams, each generation more complex, in more habitats, choosing among more alternatives. We marvel at the diversity of life in Earth's biosphere, with animals and plants and chemically agile bacteria and fungi in every nook and cranny, but the diversity and range of the post-biological world will be astronomically greater.

An ecology will arise, as individual Exes specialize. Some may choose to defend territory in the solar system, near planets or in free solar orbit, close to the sun, or out in cometary space beyond the planets. Others may decide to push on to the nearby stars. Some may simply die, through miscalculation or deliberately. There will be conflicts of interest, and occasional clashes that drive away or destroy some of the participants, but superintelligent foresight and flexibly should allow most conflicts to be settled by mutually beneficial surrenders, compromises, joint ventures or mergers. Small entities may be absorbed by larger ones, and large entities will sometimes divide, or establish seed colonies. Parasites, in hardware and software, many starting out as component parts of larger beings, will evolve to exploit the rich ecology. The scene may resemble the free-for-all revealed in microscopic peeks at pond water, but instead of bacteria, protozoa and rotifers, the players will be entities of potentially planetary size, whose constantly-growing intelligence greatly exceeds a human's, and whose form changes frequently through conscious design. The expanding community will be linked by a web of communication links, on which the intelligences barter inventions, discoveries, coordinated skills, and entire personalities, sharing the benefits of each other's enterprise.

Less restricted and more competitive, the space frontier will develop more rapidly than Earth's tame economy. An entity that fails to keep up with its neighbors is likely to be eaten, its space, materials, energy and useful thoughts reorganized to serve another's goals. Such a fate may be routine for humans who dally too long on slow Earth before going Ex. Perhaps a few will escape to expanses beyond the solar system's dangers, like newly hatched marine turtles scrambling across a beach to the sea, under greedy swooping birds. Others may pre-negotiate favorable absorption terms with established Exes, like graduating seniors meeting company recruiters--or Faust soliciting bids for his soul.

Exes will propagate less by reproduction than reconstruction, meeting the future with continuous self improvements. Unlike the blind incremental processes of conventional life, intelligence-directed evolution can make radical leaps and change substance while retaining form. A few decades ago radios changed from vacuum tubes to utterly different transistors, but kept the clever "superheterodyne" design. A few centuries ago, bridges changed from stone to iron, but retained the arch. A normally evolving animal species could not suddenly adopt iron skeletons or silicon neurons, but one engineering its own future might. Even so, Darwinian selection will remain the final arbiter. Forethought reveals the future only dimly, especially concerning entities and interactions more complex than the thinker. Prototypes uncover only short-term problems. There will be minor, major and spectacular miscalculations, along with occasional happy accidents. Entities that make big mistakes, or too many small ones, will perish. The lucky few who happen to make mostly correct choices will found succeeding generations.

Only tentatively grasping the future, entities will perforce rely also on their past. Time-tested fundamentals of behavior, with consequences too sublime to predict, will remain at the core of beings whose form and substance changes frequently. Ex-companies are likely to retain much of corporate law and Ex-humans are likely to remain humanly decent--why choose to become a psychopath? In fact, a reputation for decency has predictable advantages for a long-lived social entity. Human beings are able to maintain personal relationships with about two hundred individuals, but superintelligent Exes will have memories more like today's credit bureaus, with enduring room for billions. Trustworthy entities will find it far easier than cheaters to participate in mutually beneficial exchanges and joint ventures. In the land of immortals, reputation is a ponderous force. Other character traits, like aggressiveness, fecundity, generosity, contentment or wanderlust likely also have long-term consequences imperfectly revealed in simulations or prototypes.

To maintain integrity, Exes may divide their mental makeup into two parts, a frequently changed detailed design, and a rarely-altered constitution of general design principles--analogous to the laws and the constitution of a nation, the general knowledge and fundamental beliefs of a person, or soul and spirit in some religious systems. Deliberately unquestioned constitutions will shape entities in the long run, even as their designs undergo frequent radical makeovers. Once in a while, through accident or after much study, a constitution may be slightly altered, or one entity may adopt a portion of another's. Some variations will prove more effective, and entities with them will become slowly more numerous and widespread. Some will be so ineffective that they become extinct. Gradually, by Darwinian processes, constitutions will evolve. They will be both the DNA and the moral code of the postbiological world, shaping the superintelligences that manage day to day transformations of world, body and mind.