Table of Contents
Acknowledgments xi
Prelude: The Tale of the Omega Team 3
1 Welcome to the Most Important Conversation of Our Time 22
A Brief History of Complexity
The Three Stages of Life
Controversies
Misconceptions
The Road Ahead
2 Matter Turns Intelligent 49
What Is Intelligence?
What Is Memory?
What Is Computation?
What Is Learning?
3 The Near Future: Breakthroughs, Bugs, Laws, Weapons and Jobs 82
Breakthroughs
Bugs vs. Robust AI
Laws
Weapons
Jobs and Wages
Human-Level Intelligence?
4 Intelligence Explosion? 134
Totalitarianism
Prometheus Takes Over the World
Slow Takeoff and Multipolar Scenarios
Cyborgs and Uploads
What Will Actually Happen?
5 Aftermath: The Next 10,000 Years
Libertarian Utopia
Benevolent Dictator
Egalitarian Utopia
Gatekeeper
Protector God
Enslaved God
Conquerors
Descendants
Zookeeper
1984
Reversion
Self-Destruction
What Do You Want?
6 Our Cosmic Endowment: The Next Billion Years and Beyond 203
Making the Most of Your Resources
Gaining Resources Through Cosmic Settlement
Cosmic Hierarchies
Outlook
7 Goals 249
Physics: The Origin of Goals
Biology: The Evolution of Goals
Psychology: The Pursuit of and Rebellion Against Goals
Engineering: Outsourcing Goals
Friendly AI: Aligning Goals
Ethics: Choosing Goals
Ultimate Goals?
8 Consciousness 281
Who Cares?
What Is Consciousness?
What's the Problem?
Is Consciousness Beyond Science?
Experimental Clues About Consciousness
Theories of Consciousness
Controversies of Consciousness
How Might AI Consciousness Feel?
Meaning
Epilogue: The Tale of the FLI Team 316
Notes 337
Index 349