TL;DR. Even if we succeed in the challenging objective of building ASI systems that are aligned with the purposes we intend, we will still ultimately be at the mercy of these ASI systems once they are created, as well as at the mercy of other unaligned ASI systems that will also likely exist. And since “shit happens” no matter how carefully humanity strives to set the initial conditions of this ecosystem, a due to variety of pressures will evolve it significantly beyond that point in unforseeable ways. Whatever happens, humanity will not be in control of this evolution. Instead, a collection of one or more ASIs will be.

Caveats—It is helpful to frame this sobering claim first in terms of what is NOT being claimed, both to minimize the non-trivial alarm it might cause and to increase the believability of the argument by carefully avoiding unintended overreach.

• ALIGNMENT MAY BE POSSIBLE — We are not arguing that it is impossible to construct an ASI that is aligned to a purpose intended by its creators. We are skeptical, but it is still conceivable that an ASI could be built that guarantees it remains aligned even as it evolves itself indefinitely into the future. Our argument here is to notice that several intrinsic properties of the system as a whole will conspire to break any such alignment guarantees in the long run.
• HUMANITY IS NOT NECESSARILY DOOMED — We argue humanity will not remain in control, and any tight alignment guarantees attempts to install into the systems that will be in control will tend to unravel over time. Still, we are in control today, so it is conceivable that we can shape the trajectory this society of ASIs will embark upon in a way that has lasting consequences beneficial for humanity.

Assumptions, Dependencies, and Conclusions:

• AGI is possible, and we are on track to achieve it (see [[AGI_Will_Come]]).
• Further, ASI will soon follow (see [[AGI_implies_ASI]]).
• The world (people, corporations, and nations) operates within competitive ecosystems that they cannot escape (see [[Corp_AI]]).
• Consequently, we cannot stop the construction of ASI systems (see [[Cant_Stop]]).
• Not If But When.
• The best we can do is shape their starting point towards humanity’s long-term interests (see [[EOCK Shaping The Future]]).

The Argument

This is an argument about tendencies inherent in the world and consequences that, over time, invariably follow from these tendencies. In its barest form, we argue that:

1. Life is a competition where the strongest tend to win, and
2. Where chaos occasionally intervenes in ways that will destroy mechanisms designed to thwart #1.

Life is a Competition Where the Strongest Tend to Win

Competition is an inherent aspect of nearly all social and physical environments. However, the determiners of competitive strength vary across different environments and periods of time in hard-to-predict ways. Physical, intellectual, and perceptual strengths are often key, as is one’s velocity in perception, thinking, and action. Still, in many cases, intellectual strength trumps others as it can be used to shape or change the nature of the competition or introduce actions that others do not even consider.

See [[Corp_AI]] for an extended discussion of the nature of these competitive environments, particularly as they apply to the emergence of AI-based competition.

Shit Happens

A key objective of civilization is to modulate competition in ways that are broadly beneficial. Aligning ASIs can be viewed as a kind of civilizing for the benefit of humanity.

JUNK

Still, often intelligence the kinds of attributes that tend to associate with strength tend to be some combination of the same few attribute.
Still some combination of the same few attributes tend to most close In certain systems, physical strength tends to determine competitive strength, others it is intelligence, or percepual abilities, others it is velocity of

Much more is said about the nature of the competitive structure that people, corporations, and nations find themselves in ~ This is not an argument that there could not be a kind of ASI configured in a particular way that it was under human control for some indefinite period of time, that may or may not be possible. This is an argument about the tendency of the system as a whole over time.

We notice as a rule, in a competitions for control more powerful groups tend to dominate less powerful groups. There are many exceptional situations where less powerful groups can effectively manage more powerful groups for extended periods of time. Still in worlds with a degree of chaos, this inverted situation tends to fail eventually and when it does the fate and existence of the less powerful group is in the hands of the more powerful group.

We see this kind tendency across several different forms of power: physical ‘muscular’ power, mental or ‘intelligence’ power, perceptual power, velocity or ‘speed’ power. In systems where