Why do you care about AI Existential Safety?

AI systems control an ever growing part of our world. As a result, they will increasingly interact with each other directly, with little or no potential for human mediation. If each system stubbornly pursues its own objectives, this runs the risk of familiar game-theoretic tragedies – along the lines of the Tragedy of the Commons, the Prisoner’s Dilemma, or even the Traveler’s Dilemma – in which outcomes are reached that are far worse for every party than what could have been achieved cooperatively. However, AI agents can be designed in ways that make them fundamentally unlike strategic human agents. This approach is often overlooked, as we are usually inspired by our own human condition in the design of AI agents. But this approach has the potential to avoid the above tragedies in new ways. The price to pay for this, for us as researchers, is that many of our intuitions about game and decision theory, and even belief formation, start to fall short. Foundational research from the philosophy and game theory literatures provides a good starting point for pursuing this approach.

Please give one or more examples of research interests relevant to AI existential safety:

I direct the Foundations of Cooperative AI Lab (FOCAL) at Carnegie Mellon University. Our goal is to create foundations of game theory appropriate for advanced, autonomous AI agents – with a focus on achieving cooperation. Research directions include: Designing preferences, beliefs, and identities for artificial intelligence Open-source game theory and multilateral commitments Foundations of multi-agent learning Decision-theoretic foundations for game theory Self-locating beliefs.