Superintelligence: Paths, Dangers and Strategies by Nick Bostrom

Nick Bostrom’s Superintelligence: Paths, Dangers and Strategies inspects and dissects the topic of Superintelligence. First, laying out different possible paths to super intelligence and then discussing how these different paths might effect humanity, or might occur. It then discusses the dynamics of a singleton AI and later a multi-polar world of interacting AIs. However, Bostrom’s real point throughout this book is not just how one might build an AI. But how one might build and control an AI safely, namely the control problem (which I just found out has a subreddit dedicated to it). On one hand Bostrom suggests that human-brain emulation might lead to a super intelligence that naturally shares our values, and with proper selection, there is a small chance this solves the control problem. However, more likely it means we have no real control over what happens. On the other hand, the challenge of building an AI which is set up to be flexible enough to grow with humanity, being benevolent the whole time, and be provably this way is a major challenge in its own right.

The book was eye opening to me as an engineer. Not only does Bostrom really dig into superintelligence, but the ethics the project that designs it should have. Namely, a focus on safety and a compassionate attitude towards humanity. Also enlightening was the idea that the best way to specify most of the subsystems an AI needs might be indirectly, which at first seemed counter intuitive to me. This is do to the fact that a sufficiently smart AI (or human for that matter), if it has a goal to improve itself does not need to start life with a perfect system, as outlined by this quote:

Rather, our focus should be on creating a highly reliable design, one that can be trusted to retain enough sanity to recognize its own failings. An imperfect superintelligence, whose fundamentals are sound, would gradually repair itself; and having done so, it would exert as much beneficial optimization power on the world as if it had been perfect from the outset.

— Nick Bostrom

It was an enjoyable read and I left the book with a wonder and awe at the problems of AI. It is a little scary, but I think I agree with Bostrom’s thesis that is not if but when and I for one would rather try to contribute to a benevolent AI, than let the world get swallowed by an ignorant, sophistic or power hungry AI. I leave you with this quote:

The point of superintelligence is not to pander to human preconceptions but to make mincemeat out of our ignorance and folly.

— Nick Bostrom