Cold takes on most important century
The argument here is that even we don’t become a space faring civilization, the exponential economical growth of our species mean we live in a very pivotal time. AI or the increase of productive would create a feedback positive loop that will prompt us into a widely different future.
PASTA, or Process for Automating Scientific and Technological Advancement (guesses it will be developed this century). it allows systems to pursue their own objectives, to self replicate, etc.
AGI Safety From First Principles by Richard Ngo
The problem here is that we’re getting very good at generalized learning
Bostrom definition of superintelligence:
any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.
Perhaps the most striking is the vast difference between the speeds of neurons and transistors: the latter pass signals about four million times more quickly.
Ngo argues that AGI will figure out coordination and then become really dangerous:
The efficacy of a collective AGI might be limited by coordination problems between its members. However, most of the arguments given in the previous paragraphs are also reasons why individual AGIs will be able to surpass us at the skills required for coordination (such as language processing and theories of mind).
This is a good argument why it’s important to humans to come up with coordination solutions before machines do. We will both learn enough to understand whether AGIs are organizing themselves and learn how to use these cooperation tools to make them cooperate with us rather than with each other (and not take us into consideration).