Why do you care about AI Existential Safety?

I believe that research in AI existential safety has a good chance of reducing the likelihood that transformative AI leads to large negative outcomes (e.g. due to catastrophic or existential risks or via locking in a great deal of suffering, especially but not limited to non-human animals).

I am especially concerned with questions of governance. Our window for governing transformative AI systems could be narrow if there is significant competitive pressure to build the most capable AI systems. Good governance of AI development can ensure that the field has the time and resources they need to invent and disseminate technical and societal solutions to long-term risks.

Please give at least one example of your research interests related to AI existential safety:

My research models how to best design policies for reducing risks from transformative AI. One example of my work is the following paper: Safe Transformative AI via a Windfall Clause (arxiv.org/abs/2108.09404): This paper extends a model of transformative AI competition with a Windfall Clause (see The Windfall Clause) to show how it is possible to design a Windfall Clause that overcomes two challenges: firms must have reason to join, and their commitments must be credible. The results suggest that firms benefit from joining a Windfall Clause under a wide range of scenarios. Hence, the paper provides evidence that the Windfall Clause may be an effective tool for reducing the risk from competition for transformative AI.

I am also a member of Modeling Cooperation. We recently built a web app which implements the Safety- Performance Tradeoff model created by Robert Trager, Paolo Bova, Nicholas Emery-Xu, Eoghan Stafford, and Allan Dafoe (spt.modelingcooperation.com). The web app allows other researchers and decision-makers to explore how improvements in technical AI safety could affect the safety choices of competing AI developers.