Intelligence explosion, singularity, fast takeoff… these are a few of the terms given to the surpassing of human intelligence by machine intelligence, likely to be one of the most consequential – and unpredictable – events in our history. 

For many decades, scientists have predicted that artificial intelligence will eventually enter a phase of recursive self-improvement, giving rise to systems beyond human comprehension, and a period of extremely rapid technological growth. The product of an intelligence explosion would be not just Artificial General Intelligence (AGI) – a system about as capable as a human across a wide range of domains – but a superintelligence, a system that far surpasses our cognitive abilities.

Speculation is now growing within the tech industry that an intelligence explosion may be just around the corner. Sam Altman, CEO of OpenAI, kicked off the new year with a blog post entitled Reflections, in which he claimed: “We are now confident we know how to build AGI as we have traditionally understood it… We are beginning to turn our aim beyond that, to superintelligence in the true sense of the word”. A researcher at that same company referred to controlling superintelligence as a “short term research agenda”. Another’s antidote to online hype surrounding recent AI breakthroughs was far from an assurance that the singularity is many years or decades away: “We have not yet achieved superintelligence”. 

We should, of course, take these insider predictions with a grain of salt, given the incentive for big companies to create hype around their products. Still, talk of an intelligence explosion within a handful of years spans further than the AI labs themselves. For example, Turing Award winners and deep learning pioneers Geoffrey Hinton and Yoshua Bengio both expect superintelligence in as little as five years. 

What would this mean for us? The future becomes very hazy past the point that AIs are vastly more capable than humans. Many experts worry that the development of smarter-than-human AIs could lead to human extinction if the technology is not properly controlled. Better understanding the implications of an intelligence explosion could not be more important – or timely. 

Why should we expect an intelligence explosion?

Predictions of an eventual intelligence explosion are based on a simple observation: since the dawn of computing, machines have been steadily surpassing human performance in more and more domains. Chess fell to the IBM supercomputer DeepBlue in 1997; the board game Go to DeepMind’s AlphaGo in 2016; image recognition to ImageNet in 2015; and poker to Carnegie Mellon’s Libratus in 2017. In the past few years, we’ve seen the rise of general-purpose Large Language Models (LLMs) such as OpenAI’s GPT-4, which are becoming human-competitive at a broad range of tasks with unprecedented speed. LLMs are already outperforming human doctors at medical diagnosis. OpenAI’s model ‘o1’ exceeds PhD-level human accuracy on advanced physics, biology and chemistry problems, and ranks in the 89th percentile (the top 11%) at competitive programming questions hosted by platform Codeforces. Just three months after o1’s release, OpenAI announced its ‘o3’ model, which achieved more than double o1’s Codeforces score, and skyrocketed from just 2% to 25% on FrontierMath, one of the toughest mathematics benchmarks in the world. 

As the graphic below shows, over time AI models have been developing new capabilities more quickly, sometimes jumping from zero to near-human-level within a single generation of models. The gap between human and AI performance is closing at an accelerating pace: 

These advances in capability are being accelerated by the ever-decreasing cost of computation. Moore’s Law, an extrapolation posited by engineer Gordon Moore in 1965 that has held true ever since, states that the number of transistors per silicon chip doubles every year. In other words, computing power has been getting both more abundant and cheaper, enabling us to train larger and more powerful models with the same amount of resources. Unless something happens to derail these trends, all signs point to AIs eventually outperforming humans across the board.  

How could an intelligence explosion actually happen?

Keen readers may have noticed that the above doesn’t necessarily imply an intelligence explosion, just a gradual surpassing of humans by AIs, one capability after another. There’s something extra implied by the term “intelligence explosion” – that it will be triggered at a discrete point, and lead to very rapid and perhaps uncontrollable technological growth. This marks a point of no return, at which, if humanity loses control of AI systems, it will be impossible to regain. 

How could this actually happen? As we’ve explained, we have every reason to expect that AI systems will eventually surpass human level at every cognitive task. One such task is AI research itself. This is why many have speculated that AIs will eventually enter a phase of recursive self-improvement. 

Imagine that an AI company internally develops a model that outperforms its top researchers and engineers at the task of improving AI capabilities. That company would have a tremendous incentive to automate its own research. An automated researcher would have many advantages over a human one – it could work 24/7 without sleep or breaks, and likely self-replicate (Geoffrey Hinton has theorised that advanced AIs could make thousands of copies of themselves to form “hive minds”, so that what one agent learns, all of them do). It takes about 100 to 1000 times more computing power to train an AI model than to run one, meaning that once an automated AI researcher is developed, vastly more copies could be run in parallel.  Dedicated to the task of advancing the AI frontier, these copies could likely make very fast progress indeed. Precisely how fast is debated – one particularly thorough investigation into the topic estimated that it would take less than one year to go from AIs that are human-level at AI R&D to AIs that are vastly superhuman. Sam Altman has recently stated that he now thinks AI “takeoff” will be faster than he did previously. 

How likely is an intelligence explosion? 

This idea may sound like sci-fi – but it is taken seriously by the scientific community. In the 2023 AI Impacts survey, which features the largest sample of machine learning researchers to date, 53% of respondents thought an intelligence explosion was at least 50% likely. The survey defines an intelligence explosion as “feedback loop could cause technological progress to become more than an order of magnitude faster” over less than five years, due to AI automating the majority of R&D.

The likelihood of an intelligence explosion depends to some extent on our actions. It becomes far more probable if companies decide to run the risk of automating AI research. All signs currently point to this being the path they intend to take. In a recent essay, Anthropic CEO Dario Amodei appears to explicitly call for this, suggesting that “AI systems can eventually help make even smarter AI systems” will allow the US to retain a durable technological advantage over China. DeepMind is hiring for Research Engineers to work on automated AI research. 

Is an intelligence explosion actually possible?

There are several counterarguments one might have to the idea that an intelligence explosion is possible. Here we examine a few of the most common.

One objection is that raw intelligence is not sufficient to have a large impact on the world – personality, circumstance and luck matter too. The most intelligent people in history have not always been the most powerful. But this arguably underestimates quite how much more capable than us a superintelligence might be. The variation in intelligence between individual humans could be microscopic in comparison, and AIs have other advantages such as being able to run 24/7, hold enormous amounts of information in memory, and share new knowledge with other copies instantly. Zooming out, it’s also easy to observe that the collective cognitive power of humanity significantly outweighs that of other species, which explains why we’ve largely been able to impose our will on the world. If a Superintelligence were developed, we can expect many thousands or millions of copies to be run in parallel, far outnumbering the amount of human experts in any particular domain. 

Some also argue that an intelligence explosion will be impeded by real-world bottlenecks. For example, AI may only be able to improve itself as fast as humans can build datacenters with the required computing power to train larger and larger models. But new improvements in AI algorithms could significantly improve their performance, and make rapid acceleration possible even while the amount of computing power remains constant. Recent models are achieving state-of-the-art performance with low compute costs – for example, Chinese company DeepSeek’s R1 model is able to compete with OpenAI’s cutting-edge reasoning models, but allegedly cost less than $6 million to train. Researchers used novel algorithmic techniques to extract powerful capabilities from far less computing power. A self-improving AI would likely be able to identify similar techniques much more quickly than humans.

Another objection is that no one AI could ever create a thing superior to itself, just as no one human ever could. Instead, superintelligent AI will be a civilisational effort, requiring the combined brainpower of many humans using their most powerful tools – books, the internet, mathematics, the scientific method – not the result of recursive self-improvement. However, there have also been challenges which required the collaboration of humans over many generations, but which a single AI system overcame very quickly not by learning from human examples, but by playing against itself many millions of times. For example, the art of playing Go was passed down and developed over many generations, but DeepMind’s AlphaGo developed superhuman Go-playing abilities in days through self-play. 

Others have argued that an intelligence explosion is impossible because machines will never truly “think” in the same way as humans, since they lack certain essential properties for “true” intelligence such as consciousness and intentionality. But this is a semantic distinction that breaks down when we replace the word “intelligence” with “capability” – the concern is about whether systems can solve problems and effectively pursue goals. In the words of Dutch computer scientist Edsger W. Dijkstra, “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”

Whether an intelligence explosion is possible or likely remains unclear. Some experts, such as head of the US government’s AI Safety Institute Paul Christiano, believe that runaway recursive self-improvement is possible, but that the more likely scenario is a slow ramp up of AI capabilities, in which AI becomes incrementally more useful for accelerating AI research. This could give humanity the opportunity to intervene before systems become dangerously capable. Nonetheless, there are compelling arguments in favour of an intelligence explosion, and many scientists take the hypothesis seriously.

How close are we to an intelligence explosion?

We may not know whether an intelligence explosion is in our future until we reach the critical point at which AI systems become better than top human engineers at improving their own capabilities. How soon could this happen? 

The truth – and in many ways the problem – is that no one really knows. Each frontier AI training run produces models with emergent capabilities that can take even their developers by surprise. We could easily cross the threshold without realising. However, one effort to measure progress towards automated AI R&D is RE-bench, a benchmark developed by AI safety non-profit METR. A study published in November last year tested frontier models including Anthropic’s Claude Sonnet 3.5 and OpenAI’s o1-preview against over 50 human experts. The results showed that models are already outcompeting humans at AI R&D over short time horizons (2 hours), though they still fall short over longer ones (8 hours). OpenAI has announced a new model in the form of o3 since then, which may be better still. Results like these may be the most important indicators we have of our proximity to an intelligence explosion. As one AI researcher points out, AI research might be the only thing we need to automate in order to reach superintelligence. 

Another requirement for automated AI research is agency – the ability to complete multi-step tasks over long time horizons. Developing reliable agents is quickly becoming the new frontier of AI research. Anthropic released a demo of its computer use feature late last year, which allows an AI to autonomously operate a desktop. OpenAI has developed a similar product in the form of AI agent Operator. We are also starting to see the emergence of models designed to conduct academic research, which are showing an impressive ability to complete complex end-to-end tasks. OpenAI’s Deep Research can synthesise (and reason about) existing literature and produce detailed reports in between five and thirty minutes. It scored 26.6% (more than doubling o3’s score) on Humanity’s Last Exam, an extremely challenging benchmark created with input from over 1,000 experts. DeepMind released a similar research product a couple months earlier. These indicate important progress towards automating academic research. If increasingly agentic AIs can complete tasks with more and more intermediate steps, it seems likely that AI models will soon be able to perform human-competitive AI R&D over long time horizons. 

Timelines to superintelligent AI vary widely, from just two or three years (a common prediction by lab insiders) to a couple of decades (the median guess among the broader machine learning community). That those closest to the technology expect superintelligence in just a few years should concern us. We should be acting on the assumption that very powerful AI could emerge soon. 

Should we be scared of an intelligence explosion?

Take a second to imagine the product of an intelligence explosion. It could be what Anthropic CEO Dario Amodei calls “a country of geniuses in a datacenter” – a system far more capable than the smartest human across every domain, copied millions of times over. This could enable a tremendous speed-up in scientific discovery, which, if we succeed in controlling these systems, could deliver tremendous benefits. They could help us cure diseases, solve complex governance and coordination issues, alleviate poverty and address the climate crisis. This is the hypothetical utopia which some developers of powerful AI are motivated by. 

But imagine that this population of geniuses has goals that are incompatible with those of humanity. This is not an improbable scenario. Ensuring that AI systems behave as their developers intend, known as the alignment problem, is an unsolved research question, and progress is lagging far behind the advancement of capabilities. And there are compelling reasons to think that by default, powerful AIs may not act in the best interests of humans. They may have strange or arbitrary goals to maximise things of very little value to us (this is known as the orthogonality thesis), or the optimal conditions for the completion of their goals may be different to those required for our survival. For example, former OpenAI chief scientist IIlya Sutskever believes that in a post-superintelligence world, “the entire surface of the Earth will be covered in datacenters and solar panels”. Sufficiently capable systems could determine that a colder climate would reduce the need for energy-intensive computer cooling, and so initiate climate engineering projects to reduce global temperatures in a way that proves catastrophic for humanity. These are among the reasons why many experts have worried that powerful AI could cause human extinction. 

So, how might this consortium of super-geniuses go about driving humans extinct? Speculating about how entities much smarter than us could bring about our demise is much like trying to predict how Garry Kasporov would beat you at chess. We simply cannot know exactly how we might be outsmarted, but this doesn’t prevent us from anticipating that it would happen. A superintelligent AI might be able to discover new laws of nature and quickly develop new powerful technologies (just as human understanding of the world around us has allowed us to exert control over the world, for example through urbanisation or harnessing of energy sources). It could exploit its understanding of human psychology to manipulate humans into granting it access to the outside world. It could accelerate progress in robotics to achieve real-world embodiment. It might develop novel viruses that evade detection, bring down critical infrastructure through cyberattacks, hijack nuclear arsenals, or deploy swarms of nanobots. 

There will be great pressure to deploy AI systems whenever they promise to improve productivity or increase profits, since companies and countries that choose not to do so will not be able to compete – and this means humans might be harmed by accident. Today’s systems are unreliable and buggy, often hallucinating false information or producing unpredictable outputs, which could cause a real-world accident if it were deployed in critical infrastructure. But all these scenarios reflect our limited human imagination. Whatever we can come up with, a superintelligence could likely do one better. 

Smarter-than-human AI systems could pose existential risks whether or not they emerge from a process of rapid self-improvement. But an intelligence explosion could make them more likely. Many of the guardrails or safety measures that might work in a “slow takeoff” scenario may be rendered obsolete if an intelligence explosion is triggered at some unpredictable and discrete threshold. Implementing sensible regulation based on a common understanding of risk will be harder. Alignment techniques such as using weaker models to “supervise” stronger models (scalable oversight) or automating safety research become less viable. 

Can we prevent an intelligence explosion?

Unprecedented resources are currently being poured into the project of building superintelligence. In January, OpenAI announced a new project named Stargate, which will invest $500 billion into AI development over the next four years, with Donald Trump vowing to support its mission through “emergency declarations” that will support datacenter construction. Cash is flowing in other corners of the AI ecosystem too – Forbes has projected that US AI spending will exceed a quarter trillion this year. 

Regulation to control AI development is sparse. Governance of frontier AI largely depends on voluntary commitments, more ambitious bills have failed to pass, and Trump has already repealed Biden’s Executive Order requiring that companies notify the US government of large training runs. 

But none of this means an intelligence explosion is inevitable. Humanity doesn’t have to build dangerous technology if it doesn’t want to! There are many regulatory approaches we could take to reduce risk, from pausing development of the most powerful systems, to mandating safety standards and testing for frontier models (as was proposed in a recently vetoed bill). There are also many ideas on the table as to how we could control smarter-than-human AI, such as creating formal verifications or quantitative guarantees to ensure that powerful systems are safe. But capabilities are advancing very quickly, and just 1-3% of AI research is currently dedicated to safety. Researchers working on these critical problems need more time that meaningful regulation could help deliver. 

Read next: Keep The Future Human proposes four essential, practical measures to prevent uncontrolled AGI and superintelligence from being built, all politically feasible and possible with today’s technology – but only if we act decisively today.

Addressee: Faisal D’Souza, NCO / Office of Science and Technology Policy / Executive Office of the President / 2415 Eisenhower Avenue / Alexandria, VA 22314

About the Organization: The Future of Life Institute (FLI) is one of the US’s oldest and most influential think tanks with a focus on advanced artificial intelligence (AI). Our first research grants were funded by Elon Musk, who continues to serve as one of our external advisors. In the early days of AI policy, FLI convened industry leaders, academia and civil society to develop the world’s first AI governance framework at Asilomar in 2017. Following the launch of OpenAI’s GPT-4, our 2023 open letter sparked a global debate on the consequences of AI development for society.

Author: Jason Van Beek is FLI’s chief government affairs officer. Before joining FLI, Jason served for 20 years as a senior advisor to current Senate Majority Leader John Thune. During that time, he served in a variety of staff roles, including as Senator Thune’s staff designee to the Armed Services Committee as well as a senior staffer on the Senate Commerce Committee, and ultimately as a Senate leadership staffer. As the Commerce Committee’s top investigator, he conducted investigations of large technology companies. Jason also advised on national security, intelligence, and nuclear weapons issues when Sen. Thune served on the Senate Armed Services Committee. He can be reached at jason@futureoflife.blackfin.biz.

Executive Summary: FLI offers several proposals for safeguarding US interests in the age of advanced AI. Our submission stresses the necessity of protecting the presidency from loss of control by calling for mandatory “off-switches,” preventing the development of uncontrollable AI, and robust antitrust enforcement in the AI sector. We further highlight the importance of ensuring AI systems are free from ideological agendas, and call for a ban on models with superhuman persuasion capabilities. We also emphasize the need to protect critical infrastructure and American workers from AI-related threats, and suggest measures like export controls on advanced AI models and tracking job displacement. Finally, we propose establishing an AI industry whistleblower program alongside mandatory reporting of security incidents to foster transparent and accountable development.

---

1. Protect the presidency from power loss to an out-of-control AI or rival authority.

1.1  Issue a moratorium on developing future AI systems with the potential to escape human control, including those with self-improvement and self-replication capabilities.  

As AI systems grow more powerful, they could pose existential challenges to the presidency by enabling rival authorities or autonomous systems to undermine executive power. The emergence of superintelligent AI systems capable of recursive self-improvement poses a unique risk. These systems could potentially become uncontrollable, undermining national security. Companies are actively pursuing superintelligent autonomous systems. For example, Reflection AI, founded by veterans of landmark AI projects like AlphaGo and GPT-4, states forthrightly on its website that “ur goal is to build superintelligent autonomous systems.”

Ensuring that AI systems remain under human control is critical for maintaining US national security dominance and preventing misuse by hostile actors. Prominent figures in technology and academia, including Elon Musk, have called for caution in developing advanced AI systems more powerful than GPT-4. Advanced AI systems capable of self-improvement or self-replication could evolve beyond their original programming, making them difficult or impossible to control. A moratorium would allow time to develop robust safeguards and governance frameworks before these technologies are deployed at scale.

The presidency is a cornerstone of American democracy and must be protected from threats posed by autonomous AI systems or rival authorities empowered by advanced technologies. A targeted moratorium on developing uncontrollable AI systems is a prudent step toward ensuring that advancements in AI align with US strategic interests and values.

1.2 Ensure the US government understands and has visibility into superintelligent AI systems. 

There are important visibility and understanding functions that should be in place within the US government in order to protect national security with respect to the development of superintelligent AI systems. Key functions include regular engagement with AI labs, housing general expertise on AI, and potentially establishing an office within the National Security Council to maintain situational awareness. It will be necessary to facilitate coordination between the intelligence community, the Pentagon, industry, and civilian agencies to set up alert and emergency response mechanisms for AI threats.

1.3  Mandate installation of an off-switch for all advanced AI systems. 

The increasing autonomy and complexity of advanced AI systems necessitate proactive safeguards to mitigate risks of unintended harm. To prevent catastrophic outcomes from runaway AI systems or misuse by rival authorities, it is essential to mandate the installation of fail-safe mechanisms, or off-switches, in all advanced AI systems. An AI off-switch refers to an automatic mechanism that immediately halts an AI system’s operations when it exhibits dangerous or noncompliant behavior. It would require human intervention to be able to turn the system back on. For the most capable and autonomous systems, the off-switch should be a dead-man’s switch.

An off-switch promotes national security and national command authority resilience. Off-switches provide a fail-safe against systems that diverge from intended behaviors. Federal regulations should require all developers of large-scale AI models to integrate robust shutdown mechanisms into their systems. These mechanisms must regularly be tested to ensure functionality under various scenarios.

Mandating off-switches for all advanced AI systems is a critical step toward ensuring human control while safeguarding national security and democratic integrity. By adopting this measure, the administration can reinforce America’s global leadership in AI while mitigating risk to its governing institutions.

1.4  Require antitrust law enforcement agencies at the Department of Justice (DOJ) and the Federal Trade Commission (FTC) to engage in robust oversight and enforcement to prevent power concentration as well as market consolidation of AI development under a small handful of tech monopolies.  

The rapid development of AI technologies raises critical concerns about market concentration and anti-competitive behavior. The FTC’s January 2025 staff report on AI partnerships and investments highlights the risks posed by dominant firms leveraging their market power to distort competition.¹ To ensure that the US remains a global leader in AI innovation while safeguarding fair competition, it is imperative that competition authorities like the DOJ and FTC engage in robust antitrust enforcement against anti-competitive practices in the AI sector. Big Tech should not be permitted to hold dominant control over AI.

The FTC’s January 2025 staff report on AI partnerships and investments provides a detailed analysis of how major cloud service providers such as Alphabet, Amazon, and Microsoft have formed multi-billion-dollar partnerships with leading AI developers like OpenAI and Anthropic. The Commission voted 5-0 to allow staff to issue the report. The report’s findings underscore the need for proactive antitrust enforcement to prevent dominant firms from using their partnerships to foreclose competition or entrench their market power.

To address these challenges, the AI Action Plan should require competition authorities to strengthen merger review processes, promote transparency in partnerships, and guard against collusion. In 2021, the FTC published a report originally initiated by Chairman Joe Simons that focused on nearly a decade of unreported acquisitions by five large technology companies.² The FTC study found that these companies did not report 94 transactions that exceeded the “Size of Transaction” threshold. Transactions that exceed the size threshold must be reported unless certain other criteria are not met or statutory regulatory exemptions apply. The AI Action Plan should prioritize measures that require pre-merger notifications for acquisitions or investments involving AI companies, and evaluate whether such transactions could lead to reduced competition or create barriers for smaller players in the market. Further, the AI Action Plan should require competition authorities to guard against collusion by scrutinizing collaborative agreements between AI developers and Big Tech firms.

Unchecked consolidation in the AI sector increases risk of power loss to an out of control AI, as well as to innovation and fair competition. By directing the DOJ and FTC to prioritize antitrust enforcement against anti-competitive practices in the AI industry, the administration can safeguard innovation, fair competition, and itself.

2. Foster human flourishing from AI by promoting the development of AI systems free from ideological agendas.

2.1  Ban AI models that can engage in superhuman persuasion and manipulation. 

American-led AI development should adhere to principles of neutrality and transparency. The administration should seek to incentivize AI development that prioritizes impartiality and user autonomy. According to research³ recently published by xAI and Scale AI advisor Dan Hendrycks, AI systems exhibit significant biases in their value systems. The CEO of industry leader OpenAI, Sam Altman, has said that he expects AI to be capable of superhuman persuasion well before it is superhuman at general intelligence.⁴ Advanced AI models have the potential for creating persuasive content at scale, including synthetic media, targeted messaging, and other tools that can shape perceptions in ways that are difficult to discern or resist.

The FTC has a long-standing role in protecting consumers from deceptive and unfair practices under the FTC Act. These principles are directly applicable to emerging AI technologies that may exploit cognitive biases or manipulate public opinion. Specifically, AI models capable of engaging in superhuman persuasion present unique challenges to consumer protection and fair competition.

In accordance with EO 14179’s mandate requiring development of AI systems free from engineered social agendas, and to ensure that AI systems align with the goal of promoting human flourishing, the administration should call on the FTC to investigate AI systems that engage in superhuman persuasion. The FTC’s ability to issue Civil Investigative Demands under its compulsory process authority should be used to scrutinize AI models suspected of engaging in harmful manipulation. The administration should work with Congress to allow the FTC to impose penalties on those that develop or deploy AI systems that engage in superhuman persuasion. Ultimately, AI models that engage in superhuman persuasion and manipulation should be banned.

2.2 Require the White House Office of Science and Technology Policy and the AI & Crypto Czar to have close engagement with the White House Faith Office and all religious communities to inform the governance of AI. 

The rapid advancement of AI presents profound challenges to religious faiths and traditions. For example, OpenAI CEO Sam Altman has referred to AI as a “magic intelligence in the sky.”⁵ This phrase underscores a commonly held vision of AI as a transformative and almost divine-like force that could fundamentally reshape society. Approximately 75% of Americans identify with a traditional religious faith.⁶ Yet traditional religious perspectives are largely absent from strategic AI discussions. Traditional religions, with their experience in organizing communities and addressing existential questions, have much to offer in the AI debate.

Religious communities have historically served as moral compasses in addressing societal challenges. Faith-based organizations are deeply embedded in communities across the US, often serving as trusted intermediaries for families and individuals. Their insights can help identify real-world moral and ethical implications of AI technologies. By engaging faith leaders and organizations, AI governance can benefit from ethical frameworks that emphasize human dignity, fairness, compassion, and accountability.

The administration should place a high priority on safeguarding religious liberty in the context of the development and deployment of AI systems. The administration should protect against anti-religious bias in AI systems, and collaborate with the Department of Justice’s Civil Rights Division to monitor compliance with constitutional protections for religious freedom. These principles align with the administration’s stated goals of promoting human flourishing and avoiding ideological bias in AI systems.

In light of President Trump’s commitment to ensuring that AI development promotes human flourishing, the OSTP and the AI & Crypto Czar should collaborate closely with the White House Faith Office to integrate perspectives from religious communities into the AI Action Plan. To foster collaboration and to protect religious liberty, the administration should form a council under the joint leadership of the White House Faith Office and the OSTP, composed of representatives from religious traditions. This council should be tasked with advising on ethical guidelines for AI development, deployment, and use.

3. Protect American workers from job loss and replacement.

3.1  Task the Secretary of Labor with tracking AI’s potential to replace workers, including a breakdown of the impact across different states. 

AI is transforming industries and reshaping the workforce at an unprecedented pace. While AI promises economic growth and innovation, it also poses significant risk to American workers, particularly in terms of job displacement and regional economic disparities. Moreover, while conventional AI displaces specific tasks, Artificial General Intelligence (AGI) presents a fundamental job replacement paradigm. AGI systems capable of human-level reasoning will not merely displace roles, but replace entire job categories through exponential improvements in accuracy, scalability, and cost efficiency. Therefore, the administration’s AI Action Plan represents an opportunity to confront the transformative workforce impacts of AGI. To ensure that the benefits of AI are distributed equitably and that workers are not left behind, the AI Action Plan should task the Secretary of Labor to develop a comprehensive plan to track and mitigate AI’s impact on employment across the US.

The Secretary of Labor should be directed to establish and resource a national workforce monitoring initiative to assess the impact of AI on jobs. As part of this initiative, the Secretary of Labor should regularly provide a state by state impact analysis of AI’s effects on employment, and how new AI technologies are affecting employment levels. The Secretary of Labor should create detailed reports highlighting which sectors are most at risk in each state, and identify states requiring urgent intervention due to high vulnerability.

4.  End the free giveaway of US frontier AI technology to adversaries.  

The United States should view frontier AI models (Regulated Export Systems with Top-tier Risk Implications for Critical Technology, or ‘RESTRICT’ models) as one of its most critical assets. By leading in this technology, we can promote human flourishing while maintaining global dominance.

However, too often, RESTRICT models are freely given away. There is therefore a small category of AI systems which, when shared with adversaries, would represent unacceptable transfer of national intelligence, potentially enabling terrorists to build a bioweapon, for example.

Of course, not all models are dangerous. For this reason, an expert body like the National Institute for Standards and Technology (NIST) should create red lines around precisely which models should be subject to an export control. To make that determination, they should take into account:

1. Potential for enabling catastrophic cyberattack capabilities: Advanced AI systems can dramatically accelerate the discovery of zero-day vulnerabilities and automate the development of sophisticated attack vectors against critical infrastructure. Such capabilities in the wrong hands could enable unprecedented cyberattacks against power grids, financial systems, or military command structures with potential for widespread societal disruption.
2. Risks of enhancing terrorist capabilities in domains including, but not limited to, biological weapons development: AI systems that can rapidly design novel molecules or efficiently analyze genetic sequences could lower the expertise barrier for non-state actors seeking to develop bioweapons. These technologies could enable terrorist groups to develop threats that have traditionally required state-level resources and expertise, potentially creating asymmetric threats that are difficult to anticipate or counter.
3. Comprehensive threat assessments from the intelligence community: Our intelligence agencies possess unique insights into the capabilities and intentions of foreign adversaries that must inform any policy on AI export controls. Their assessments can identify which specific AI capabilities would most significantly enhance adversarial military or intelligence operations, allowing for targeted restrictions rather than blanket limitations on technological development.

Furthermore, for export controls on RESTRICT models to be effective, there should not be an open-source exception. While open-sourcing can help promote research on a global level, this administration should consider the potential for Big Tech to inadvertently arm adversaries of the United States when it open-sources its most powerful models.⁷ 

While Big Tech companies like Meta have said that dangerous uses of their products are prohibited by their terms of service⁸, the reality is that terms of service are unlikely to dissuade any foreign agents from taking advantage of these assets to destabilize U.S. national security. As such, the U.S. government should be equipped with fail-safes within the chips powering RESTRICT systems.

Therefore, the forthcoming executive order should:

1. Require continuous affirmative licensing for chips powering RESTRICT models: Mobilize BIS to implement a licensing system in clusters of advanced AI chips exceeding a defined computational threshold. This should be akin to a dead-man’s switch, requiring regular renewal of licensing via signals from authorized servers, automatically disabling functionality if these verification checks fail or if the chip detects operation outside approved geographic boundaries. The implementation would include tamper-resistant security modules to prevent circumvention of these controls by adversaries.
2. Ensure RESTRICT chips have geolocation capabilities: Require that hardware providers incorporate secure, encrypted communication channels that allow for geolocation and geofencing to detect if a RESTRICT model is being being deployed by a foreign adversary.

5. Condition privileged energy grid access for AI companies on verifiable security measures to prevent foreign theft. 

The protection of AI systems integrated into our national energy grid against foreign theft represents a critical national security imperative that warrants executive action. As adversarial nations increasingly target US critical infrastructure through sophisticated cyber operations, our electrical grid—with its newly integrated AI systems—presents an appealing target. Foreign actors who successfully compromise AI-enhanced grid systems could not only steal valuable intellectual property but potentially manipulate grid operations, causing widespread disruptions, economic damage, or even physical harm to Americans.

The Stargate project will provide AI companies with an unprecedented amount of energy to power their technologies.⁹ This investment has to be protected. Advancements in AI capabilities create new cyber vulnerabilities, as compromised AI systems could enable more sophisticated attacks than conventional software. By conditioning privileged grid access on verifiable security measures, the Executive Order would create powerful incentives, enforced by the Department of Energy, for implementing comprehensive protections established by BIS.

6. Establish an AI industry whistleblower program to incentivize AI development that is free from ideological bias or engineered social agendas and promotes national security.

6.1  Direct the AI Czar to coordinate with Congress to establish an AI-specific whistleblower program to report dangerous signs of AI control loss or negligent practices that threaten the American people and strengthen our adversaries.  

The risk of ideological biases or engineered social agendas infiltrating AI systems poses a significant threat to public trust, societal stability, and American interests. As the US seeks to sustain and enhance its global leadership in AI, it is critical to ensure that the AI industry operates with integrity, transparency, and accountability. It is in the public interest to report dangerous signs of AI control loss, negligent practices, or development of systems that promote ideological bias or engineered social agendas. To achieve this, the AI Action Plan should form a working group to coordinate with Congress on the establishment of an AI-specific whistleblower program that incentivizes individuals to report wrongdoing. This program will bolster public trust in AI technologies while safeguarding national security, economic competitiveness, and ethical innovation.

The program should include a secure, user-friendly platform for submitting confidential reports of wrongdoing. It should also include robust legal safeguards, such as anonymity options and protections against retaliation, to ensure whistleblowers can come forward without fear. The program will hold developers and organizations accountable, deterring unethical behavior and promoting a culture of responsibility within the AI industry. In the global race for AI supremacy, responsible AI development is a strategic advantage. An AI industry whistleblower program reflects our values: freedom, innovation, fairness, transparency, and accountability.

The establishment of an AI whistleblower program is a critical step toward ensuring that AI development in the US remains responsible, transparent, and free from ideological biases or engineered social agendas.

6.2  Require NIST to issue instructions to companies on what security incidents must always be reported. 

Engaging with AI developers and deployers, NIST should develop and issue guidelines for companies on AI-related security incidents that must always be reported. Key components of these guidelines should include defined categories of security incidents that require mandatory reporting, including unauthorized access to AI systems or training data, detected vulnerabilities in AI models that could lead to exploitation, AI-generated or AI-amplified cyber threats, and incidents involving AI systems in critical infrastructure.

NIST should establish a 72-hour reporting requirement for critical incidents, and develop a standardized reporting form and secure online portal for submitting AI incident reports to relevant government agencies. Companies must report, at a minimum, a description of the incident and its impact, affected AI systems and data, potential consequences and mitigation efforts, and indicators of compromise.

---

This document is approved for public dissemination. The document contains no business-proprietary or confidential information. Document contents may be reused by the government in developing the AI Action Plan and associated documents without attribution.

---

Notes & References

^{↩ 1} FTC Staff Report on AI Partnerships & Investments 6(b) Study, January 2025.

^{↩ 2} Non-HSR Reported Acquisitions by Select Technology Platforms, 2010-2019: An FTC Study, September 2021.

^{↩ 3} See this tweet and full paper M. Mazeika et al., “Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs,” arXiv preprint arXiv:2502.08640 (February 2025).

^{↩ 4} Sam Altman (@sama): “i expect ai to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes”, 24 October 2023, link.

^{↩ 5} OpenAI Chief Seeks New Microsoft Funds to Build ‘Superintelligence,’ Financial Times, November 13, 2023.

^{↩ 6} How Religious are Americans?, Gallop, March 2024.

^{↩ 7} For a useful discussion of compute governance and international security, see “Interim Report: Mechanisms for Flexible Hardware-Enabled Guarantees,” August 23, 2024, by James Petrie, et. al.

^{↩ 8} Exclusive: Chinese researchers develop AI model for military use on back of Meta’s Llama, Reuters, November 2024.

^{↩ 9} Trump announces private-sector $500 billion investment in AI infrastructure, Reuters, January 2025.

Ann is a Senior Adviser on Autonomous Weapons Systems Governance at FLI. She has over 20 years of experience in driving multilateral governance strategies and shaping humanitarian policies at the UN, and in corporate roles at Fortune 500 companies, where she partnered with business leaders to drive inclusion in the workplace. Previously, she held roles as Deputy Permanent Observer at the International Committee of the Red Cross’s Delegation to the United Nations and Vice President at Credit Suisse. Additionally, she collaborated with civil society and local government on community economic development in Mauritania with the U.S. Peace Corps. She is a graduate of Rutgers Business School in Business Management.

This post comes from a member of our AI Existential Safety Community.

Artificial Intelligence (AI) is rapidly transforming various sectors, and education is no exception. As AI technologies become more integrated into educational systems, they offer both opportunities and challenges. This article will focus on the strategic decisions that must be considered for this technology in educational settings, as well as the impact it is having on children considering both what we should be teaching in a future where it is embedded throughout society and the shorter term risks of students using tools such as Snapchat’s ‘My AI’ without knowledge of data and privacy risks.

This blog post explores the impact of AI in education, drawing insights from recent conferences and my experiences as an AI in Education consultant in the UK. 

AI in Education: A Global Perspective 

In early September, I attended the UN Education Conference in Paris. The summit highlighted both the positive and negative effects of AI and discussed ways to mitigate and support these changes. 

Countries like Australia and South Korea have been particularly successful at integrating AI technologies into real-world applications. For instance, many Australian states have developed large-scale Copilots built upon Azure data sets, which provide strategic support for learners while collecting new data as they use educational chatbots (a step towards “personalized” learning). This is an important stepping stone for education; in the past data sets have been held by schools individually, whereas this collective method of holding and using data on a strategic level opens more opportunities for targeted applications. However, it also increases vulnerability to cyber threats and the broader impacts of AI misalignment if not monitored thoroughly.

During the conference, Mistral AI and Apple highlighted the major differences between AI models based on how they are trained. Mistral emphasized that their model has been trained to be multilingual, rather than originally trained in English and then translated. This has a significant impact on the learning interface for students and underscores how subtle differences in training data or techniques can have major impacts on educational outcomes. Education institutions across the world must consider which tools are best aligned to their students’ outcomes and unique learning needs, rather than making rash decisions to acquire new technologies that are misaligned with their values. 

Collaboration between Educators & Policy Makers to shape the Future of Education 

Although the guidance which has been produced by UNESCO for AI in education is brilliant, there is a disconnect between those who write policy for the education sector and those who are actually involved in education. During my time in Paris, I observed that many educators are already well-versed in using AI tools, while policymakers were in awe of basic custom chatbots like those built in Microsoft Copilot Studio, Custom GPTs, or Google Gems. This concerns me as, while technology in the education sector is often poorly funded, the technical and pedagogical understanding of AI’s potential in education is mostly held by those working in schools. 

There must be more opportunities for alignment between educators and government moving forward. Stakeholder meetings that include experts in education should not be limited to university researchers, but also include teachers who are using these tools regularly. Without this, governance will continue to miss out on the detailed knowledge of those in the education sector and simply rely on statistical research or the interpretation of researchers. 

Deepfakes 

Another concern highlighted in Paris was the rise of deepfakes. I have seen the impact they’ve had in schools in South Korea and the UK, with students easily generating deepfakes of their peers and teachers. This is an area that must be addressed by policymakers. Many schools are trying to tackle this “in house” through wellbeing programs, while other schools have taken a more innovative approach. In China for example, they have created deepfakes of their headteachers to promote an understanding of the potential risks and harms of the technology. DeepMind’s development of SynthID, featured in the journal Nature, may help increase the transparency of AI-generated content. 

Innovative Tools and Ethical Considerations 

Whilst working on an advisory panel with Microsoft, aiding in their development of AI tools grounded in pedagogy and centered around safety for users, I have witnessed a range of innovative tools that will transform the education landscape. One of the most significant developments is Microsoft Copilot 365, which is set to revolutionize teaching practices. This product can mark assignments, create lesson resources, and conduct data analytics within the 365 suite, which many schools already use. I believe this product has the potential to solve the global teaching shortage by streamlining the administrative elements of education, allowing teachers to focus solely on pedagogy and pastoral care. 

However, there are concerns regarding the dominance of Microsoft and Google in the education space. Although both companies have vast educator support and input, there are questions about transparency and ethical considerations in their products’ design. Additionally, many student-facing chatbots are designed for rote learning, yet we live in a time where creativity and critical thinking are more important than ever. This highlights the need for more governance and oversight in the space of AI in education. 

Data and Cybersecurity 

Data and cybersecurity are significant concerns in education. Although many schools have moved to cloud computing to make systems more secure, the centralization of data by governments and councils, such as in Australia, creates enticing targets and vulnerabilities for hackers. 

A further concern associated with this data, which includes emotional, social, and medical information, is how it is used by companies and AI tools themselves. Microsoft explicitly states that this data is not used by Microsoft and is not communicated to OpenAI, a company which has been slow to report its own data breaches. Google does not provide a clear explanation of how its education data is held. 

We must also raise the question of how we can ensure this information is not creating biased outputs. Could the echo chamber of a student’s “personalized” chatbot actually be altering their worldview in detrimental ways? The more personalized these models become for students, the higher the risk. 

Although AI usage has been well measured in adults by companies such as Microsoft and OpenAI, how young people use generative AI is not well understood globally. Snapchat’s MyAI has been used by over 150 million users so far. The majority of this user base consists of young people in the USA & UK. Some education studies also show wide scale use of ChatGPT by young people, but without the commercial data protection offered by certain organizations, and given Snapchat’s lack of adequate privacy controls, data protection and content filters, there is growing concern in the education community about the impact this could have.

To address this, we need parent consultation and digital safety education. These companies must also be held accountable for allowing children to access AI tools without suitable privacy and safety controls. This has not been the priority of all governments, companies and policy makers so far in AI development, with economic growth being prioritized over children’s safety. This is demonstrated by the veto of the SB 1047 Safety Bill, a bill which was proposed in California aimed at regulating advanced AI systems to prevent catastrophic harms. It would do this by requiring developers to conduct risk assessments, implement emergency shutdown mechanisms, and include whistleblower protections. Governor Gavin Newsom vetoed the bill on September 29, 2024, arguing that it might give a false sense of security by focusing only on large-scale models and potentially stifle innovation. This veto highlights that children’s digital safety is not a priority of those in governance and so must become a deeper priority for those working with young people. 

A Curriculum for the Future 

Beyond the practical deployment of AI in education, there begs the question as to what we should be teaching young people today. This question has been raised by many, with answers provided by The Economist, World Economic Forum, and Microsoft. However, education must go beyond the economic focus that the sector has today with curricula centered around economic growth, and look to educate children in a holistic way that will enable them to thrive in the future. UNESCO provides an outstanding guideline for this, as shown below: 

Clifton High School in the UK has launched a course tailored to execute this framework, with students learning fundamental skills such as emotional intelligence, citizenship, innovation, and collaboration through different cultures, as well as learning about AI. This “Skills for Tomorrow course aims to provide a proof-of-concept for schools across the world to replicate, providing an education that is complementary to the changes we are seeing in the 21st century. It is vital that governments and policymakers take action to reduce the risk of redundant skills in the next generation. 

Access and Socio-Economic Divides 

A global issue in education that exacerbates socio-economic divides is the question of access. With 2.9 billion people still offline and major technological divides within countries, those who can and are able to learn how to use AI in schools will benefit from these tools, while those who cannot are left behind. During my time at the UN however, I was made aware of promising projects in this area. One example is the AU Continental Strategy for Africa, which aims to focus education in African countries on capturing the advantages of artificial intelligence. It hopes to harness the adaptability of Africa’s young population, with 70% of African countries having an average age below 30. I also saw AI used to support social and emotional learning in Sweden and Norway. 

However, it has been highlighted by Everisto Benyera in their report titled “The Fourth Industrial Revolution and the Recolonisation of Africa” that strategies such as the AUCSA could be hijacked by technology companies and lead to data and employment colonization, leading to asymmetrical power relations (unevenly distribution of resources and knowledge) and further challenges for global equality in the education sector. 

Conclusion 

There is a lot to be optimistic about as AI becomes integrated into education, but the pace at which it is integrated could lead to detrimental outcomes due to improper governance, outdated national curriculums, and teachers themselves not being ready. Action must be taken, and not simply theorized, by governments to prepare the education sector globally for this technology. Its rapid development is even leaving edtech experts chasing a train of innovation.

Positive legislative efforts, such as the EU AI Act built upon the EU GDPR Law, will mean that safety is at the forefront of further testing on AI and that steps are taken to make sure that the data of children is not misused. While companies abiding by such laws should be praised, those that are designing education tools with safety as their secondary objective must realign their values. 

Jason Van Beek is FLI’s Chief Government Affairs Officer. Previously, he was General Counsel for Senate Majority Leader John Thune. Jason holds a J.D. from the University of South Dakota School of Law and a master’s degree in strategic studies from the Naval War College. He lives in Virginia with his wife and children.

Introduction

The past decade has seen the extraordinary development of artificial intelligence from a niche academic pursuit to a transformative technology. AI tools promise to unlock incredible benefits for people and society, from Nobel prize-winning breakthroughs in drug discovery to autonomous vehicles and personalized education. Unfortunately, two core dynamics threaten to derail this promise:

1. First, the speed and manner in which AI is being developed—as a chaotic nearly-unregulated race between companies and countries—incentivizes a race to the bottom, cutting corners on security, safety and controllability. We are now closer to figuring out how to build general-purpose smarter-than-human machines (AGI) than to figuring out how to keep them under control.
2. Second, the main direction of AI development not toward trustworthy controllable tools to empower people, but toward potentially uncontrollable AGI that threatens to replace them, jeopardizing our livelihoods and lives as individuals, and our future as a civilization.

With many leading AI scientists and CEOs predicting AGI to be merely 1-5 years away, it is urgent to correct the course of AI development. Fortunately, there is an easy and well-tested way to do this: start treating the AI industry like all other high-impact industries, with legally binding safety standards, incentivizing companies to innovate to meet them in a race to the top. We make a concrete proposal for such standards below.

The Need to Act Now 

Only six years ago, many experts believed that AI as capable as GPT-4 was decades, even centuries away. Now OpenAI’s o3 system recently scored 85% on the ARC-AGI benchmark, and showcases human-level reasoning skills and PhD level skills in biology, chemistry and physics. OpenAI’s “Deep research” feature scores 18% on the ultra-difficult “Humanity’s Last Exam“. Its CEO Sam Altman has claimed they already know how to build AGI, and other companies have explicitly or implicitly announced it as a goal. It is imperative then to understand what AGI is and what it would mean to develop it on our present path.

AI Risk Thresholds and AGI’s Three Parts  

To avoid innovation-stifling governmental overreach, we recommend classifying AI systems into tiers of increasing potential risk, with commensurately stricter standards, akin to the five-tier U.S. classification of drugs, and the “AI Safety Levels” from Anthropic’s Responsible Scaling Policy. As described below, our risk tiers are grounded in the capabilities of AI systems in the three core areas required for AGI.

Although AGI stands for “Artificial General Intelligence”, it is a useful mnemonic to think of it as an “Autonomous General Intelligence”: the triple combination of Autonomy (independence of action), Generality (breath of capability, application, and learning), and Intelligence (competence at tasks). AI systems possess these characteristics in different measures.

While possession of these characteristics can deliver enormous rewards from AI, high levels of convergence between them can result in correspondingly high degrees of unpredictability and risk, with the most dangerous and uncontrollable systems possessing high levels of all three. An AI with all these characteristics would be capable of the large range of effective cognition and action that a human is. However, it would be much more capable—and dangerous—than any individual human, given its scalability in speed, memory, and reproducibility, and its potential for rapid self-improvement and replication.

The combination of all three characteristics is currently unique on Earth to homo sapiens. This is why possession of all three capabilities could enable machines to replace us, as individuals in our work or in a wider sense as a species. This is what has made AGI both a target of development and an unprecedented risk.

AGI: Autonomous Generally Intelligent Systems to Fully Replace Humans

The nature of AGI as a human replacement—rather than tool—is implicit in how it has been traditionally been defined: as a system that can match human performance on most intellectual tasks. Even more tellingly, OpenAI has defined AGI as “highly autonomous systems that outperform humans at most economically valuable work”.

The desire to build powerful machines that can entirely replace humans as workers, educators, companions and more is the goal of a tiny minority. How can we ensure that we instead follow the widely desired path—one that makes us strong, rather than obsolete?

Tiered Safety Standards 

Our proposed AI risk tier classification counts the number (between 0 and 3) of A, G and I factors that are present to high degree within a given AI system.

For example, Google DeepMind’s AlphaFold, which solved the protein-folding problem, is more intelligent than any human at its (narrow) task, but not autonomous or general. It therefore scores 1, placing it in Risk Tier 1. OpenAI’s GPT-3 was very general (it could answer questions, generate text, assist with basic coding etc.) but it was not very competent (it was inaccurate, inconsistent, and reasoned poorly). Therefore it falls squarely into Tier 1. Their recently released o3 model, however, has demonstrated human-level reasoning and can answer PhD level science questions, while still being very general, so is in Risk Tier 2.

The diagram above us allows us to identify and categorize systems with different levels of A/G/I convergence (and therefore risk). This categorization allows us to place systems in corresponding risk tiers (see the table below), to which a corresponding level of safety and controllability requirements can be applied: roughly speaking, further from the center corresponds to lower risk. Different tiers of convergence trigger different requirements for training (e.g. registration, pre-approval of safety plan) and different requirements for deployment (e.g. safety cases, technical specifications).

This approach avoids overly onerous requirements being placed on relatively low-risk/narrow systems, while ensuring that controllability can be guaranteed for more potentially dangerous ones. Rather than hindering competition, this tiered approach will drive companies to innovate to meet requirements, helping to realize the incredible benefits of AI in a secure, responsible, and intentional way.

This approach does not require any re-imagining of industry governance. Companies in any other sector, from automobiles and pharmaceuticals to sandwiches, must provide satisfactory evidence that their products are safe before release to the public. They must meet safety standards, and the AI industry should be no different. Furthermore, it is sensible and consistent to place higher standards on technologies or products that have greater potential for harm. We would not want nuclear reactors tested in the same category as sandwiches.

Risk Tier	Trigger(s)	Requirements for training	Requirements for deployment
RT0	AI weak in autonomy, generality, and intelligence	None	None
RT1	AI strong in one of autonomy, generality, and intelligence	None	Qualitative guarantees: Safety audits by national authorities wherever the system can be used, including blackbox and whitebox red-teaming
RT2	AI strong in two of autonomy, generality, and intelligence	Registration with national authority with jurisdiction over the lab	Quantitative guarantees: National authorities wherever the system can be used must approve company-submitted assessment bounding the risk of major harm below authorized levels
RT3	AGI strong in autonomy, generality, and intelligence	Pre-approval of safety and security plan by national authority with jurisdiction over the lab	Formal guarantees: National authorities wherever the system can be used must certify company-submitted formal verification that the system meets required specifications, including cybersecurity, controllability, a non-removable kill-switch, and robustness to malicious use
RT4	Uses more than 1027 FLOP for training or more than 1020 FLOP/s for inference	Prohibited pending internationally agreed lift of compute cap	Prohibited pending internationally agreed lift of compute cap

Safety Guarantees and Controllability  

Ensuring safe and controllable systems requires safety standards that scale with a system’s capabilities, all the way up to and including AGI—which might soon develop into superintelligence. Risk Tiers 0, 1, 2 and 3 correspond to systems that are progressively more difficult to control, and whose potential harm is progressively greater. This is why the corresponding requirements in the table are progressively stricter, ranging from none to qualitative, quantitative and formal proofs. We want our tools to be powerful but controllable (who wants an uncontrollable car?), so we define “Tool AI” as AI that can be controlled with an assurance level commensurate with its Risk Tier.

Systems with a low degree of risk should not receive onerous requirements that limit their positive impact. Tier 0 systems are therefore unregulated, while Tier 1 systems require only qualitative safety standards. Since Tier 2 systems have significantly greater potential for harm, they require quantitative guarantees, just as is currently the case for other industries with powerful technology. For example, U.S. nuclear power plants are only allowed if government-appointed experts approve a company-provided study quantifying the annual meltdown risk as less than one in a million; similarly, the FDA only approves drugs whose side effects are quantified below an acceptable level. There are many promising approaches to providing such quantitative AI safety guarantees (Dalrymple et al. 2024).

Tier 3 (AGI) is much more risky because it can broadly match or exceed human ability, deceive people, create long-term plans, and act autonomously in pursuit of goals—which by default include self-preservation and resource acquisition. This is why top AI researchers have warned that AGI may escape human control and even cause extinction. To guarantee that AGI remains controllable, a Tier 3 system must be mathematically proven to be controllable using formal verification—which includes proving that it will never resist being turned off.  

Just as AI progress has revolutionized our ability to auto-generate text, images, video and code, it will soon revolutionize our ability to auto-generate code and proofs that meet user-provided specifications. In other words, rather than deploying unverifiable black-box neural networks, it may soon be possible to have AI systems write deployable formally verifiable code, implementing powerful machine-learned algorithms and knowledge (see Tegmark & Omohundro 2023, and provablysafe.ai for an overview of the field).

Frequently Asked Questions 

Q: But isn’t powerful AI science fiction, or decades away?

No. Six years ago, most scientists thought language-mastering AIs like GPT-4 were decades away; now they are commonplace. Today’s AIs have already arguably passed the Turing Test. Timelines have collapsed, with tech CEOs like OpenAI’s Sam Altman, Anthropic’s Dario Amodei and DeepMind’s Demis Hassabis predicting that AGI will be built in 1-5 years, and many whistleblowers, investors and academics concurring.

Q: But isn’t AGI necessary to reap AI’s benefits?

No. Almost all of the benefits cited by those seeking to build AGI can be reliably captured by intentionally developing Tool AI systems to solve specific problems. Tool AI can save millions of lives per year on roads, greatly improve cancer diagnosis, and realize breakthroughs in pandemic prevention, education, energy reduction and more. It has already helped us fold proteins and develop new medicines, through the Nobel Prize-winning AlphaFold.

Q: But surely AGI is controllable?

No. We are closer to building AGI than we are to controlling it. In fact, we have no idea how to control it. As Alan Turing presciently observed in 1951, “once the machine thinking method had started, it would not take long to outstrip our feeble powers… We should have to expect the machines to take control.” When it comes to intelligent entities, the smarter ones tend to take control—just like humans did. It doesn’t matter if the AI is evil or conscious, only that it is extremely competent, and accomplishes goals that aren’t aligned with our own. Companies building AGI have produced zero evidence of having solved the control problem.

Q: But isn’t uncontrollable AGI desirable?

No. There are people who want to deny humans their right to a meaningful future, and want us to be replaced by machines through uncontrollable AGI, but they are a tiny minority. Everyone else is firmly on “Team Human”, and would prefer that we keep control of our destiny.

Q: But surely AGI is inevitable?

No. There are many powerful, profitable technologies that we have successfully banned because we decided they were too dangerous—from human cloning to bioweapons. All we need to do to course-correct is enact safety standards that require AI companies to guarantee their products are safe before releasing them, just like in every other industry.

Q: But won’t a country imposing AI standards get overtaken?

No. Uncontrollable AGI presents the greatest threat to the country building it, more so than any adversary. Therefore any supposed advantage attained by building it first would be incredibly short-lived. Once countries recognize this danger, they will not only impose their own domestic standards to prevent uncontrollable AGI, but will also work together to prevent other countries from doing building it. Furthermore, the proposed safety standards will drive unprecedented innovation, as we use AI itself to build incredibly powerful Tool AI that we can reliably verify and trust—securely realizing the benefits of this amazing technology.

Q: Why aren’t voluntary industry commitments enough?

Because companies are stuck in a race to the bottom whose incentives favor them abandoning such commitments rather than getting outcompeted.

See our work on the AI Safety Summits, including recommendations for the upcoming AI Action Summit and our engagement on previous summits.

Technology background

Recent developments in AI, notably the o3 model from the US company OpenAI, demonstrate a worrying acceleration in capabilities. Recent benchmarks (ARC-AGI, CodeForce, GPQA) indicate that the latest models are now outperforming human experts in many critical areas. This rapid evolution, combined with the increasing commercialization of autonomous AI agents, creates systemic risks requiring urgent, coordinated international action.

History of international AI summits

• 1-2 November 2023, UK: Bletchley Park AI Safety Summit 2023
• 21-22 May 2024, South Korea: Seoul AI Safety Summit 2024

Achievements of previous summits

• Development and presentation of an (interim) international scientific report on the safety of advanced AI, to create a shared scientific understanding of the risks of general-purpose AI systems.
• Creation of a non-binding minimum standard for companies developing these systems by defining thresholds at which the risks of a general-purpose AI system become intolerable in the absence of mitigation measures (Frontier AI Safety Commitments).
• Creating a role for governments to participate in safety testing, and creating AI Safety Institutes and an international network for them.

Ambitions and deliverables for the Paris Summit

At this stage, the Summit week is planned as follows:

• February 6-7: Scientific days in Saclay to provide diplomats with a scientific overview to serve as a basis for discussions at the Summit on February 10 and 11.
• February 8-9: AI Cultural Days in several French cities.
• February 10-11: AI Action Summit
  • February 10: Multilateral exchanges with thematic round tables reserved for leaders.
  • February 11: Meeting of heads of state and government at the Grand Palais, with a parallel business event at Station F.

In terms of deliverables, we can expect announcements on:

• The creation of an AI foundation to equip developing countries with open-source AI tools based on less powerful AI systems, funded to the tune of 2.5 billion euros over 5 years.
• The presentation of 35 “convergence challenges” to showcase the impact of current AI systems in sectors such as healthcare and climate change adaptation. 
• A multilateral agreement on the environmental impact of this technology is expected to be signed at the end of the Summit.
• The completed ‘International AI Safety Report‘: 100 independent AI experts from around the world have released the first-ever International AI Safety Report, and it is expected to be presented in Paris. Backed by 30 countries and the OECD, UN, and EU, the Report summarises the state of the science on AI capabilities and risks, and how to mitigate those risks. It will help to further a common understanding of the risks posed by general-purpose AI systems.

View the official Summit webpage.

Ima Bello, Head of Summits for the Future of Life Institute (FLI)