1) What are the main components (i.e., inputs) necessary to build, train, deploy and distribute generative AI systems? Please explain the importance of these components

Generative AI (GAI) systems are the user-facing applications built on top of general purpose AI (GPAI) models. These models undergo training and inference using cloud computing, typically infrastructure as a service (IaaS), and advanced semiconductors. ¹ This requires access to limited and expensive chips, cloud capacity, based on an extensive volume of servers, vast amounts of (high quality) data, and sought-after skills needed to develop competitive GPAI models, including discovering innovative algorithms that advance the state-of-the-art.

Currently, buying API access to these models, or downloading open source alternatives, to adapt them into generative AI applications is much cheaper and faster than building these models in-house from scratch.² This means that only a handful of the most well-resourced corporations in history can afford to bankroll the development of the models that structure and underpin the applications upon which they are built. The generality of these models in competently performing a wide range of distinct tasks means that they could quickly become the digital infrastructure that forms the bedrock of the entire economy.³ 

GPT-4 boasts an end-user base of 100 million weekly active users and a business user base of over two million developers using it as a platform, including 92% of Fortune 500 companies.⁴ OpenAI’s GPT store allows users to develop and monetise their own GPTs, illustrating the base layer infrastructural nature of their GPAI model.⁵ These corporations are also preeminent in other markets, allowing them to disseminate GPAI models across cloud, search, social media, operating systems, app stores, and productivity software.⁶ Thus, the implications of market concentration are much starker than for other technologies. The combination of concentrated development resources with ubiquitous adoption and distribution throughout adjacent markets risks a winner-take-all scenario, as explored in this feedback.

2) What are the main barriers to entry and expansion for the provision, distribution, or integration of generative AI systems and/or components, including AI models? Please indicate to which components they relate.

Increasing the number of model parameters enhances a model’s capabilities by improving its capacity to learn from data. However, this requires more computing power, in chips and cloud capacity, and data, which makes it cost prohibitive for many SMEs and startups.⁷ The market for chips, the first layer in the AI value chain, is highly concentrated, a phenomenon which is exacerbated by shortages stemming from significant demand-supply imbalances in components. General purpose AI is fuelled by the parallel computation processing capabilities of NVIDIA-designed graphic processing units (GPUs), which capture 90% of the market⁸, and are manufactured by Taiwan Semiconductor Manufacturing Company (TSMC), which, in turn, captures the largest share in the global chips foundry market at 56%⁹. Many developers train AI models in CUDA, NVIDIA’s proprietary software development platform, but they must use NVIDIA’s GPUs.¹⁰ Even the well-capitalised challengers in this market observe competition issues, as OpenAI’s CEO, Sam Altman, has sought to raise $5-7 trillion to create his own chip-building capacity, highlighting the difficulties of competing on chips.¹¹

While the hardware market in semiconductors is almost a monopoly, the infrastructure market is more like an oligopoly¹², which should still concern the Commission from a competition perspective.¹³ Amazon’s AWS (31%), Microsoft’s Azure (24%) and Google Cloud (11%) collectively cover two thirds of the cloud computing market.¹⁴ This collective dominance arises from the significant investment required to establish data centres, server farms, and the network infrastructure to interconnect them.¹⁵ If OpenAI, Anthropic or DeepMind were to create their own in-house cloud infrastructure, independent of the Big Tech companies that have partnered, merged, or acquired them, it would require considerable investments in land, energy, and datacentre equipment (cabling servers, server racks, coolers, etc.).¹⁶ While the Data Act may abolish egress charges, excluding those related to parallel data storage, there remain additional non-financial hurdles hindering GPAI model developers from establishing their own in-house cloud hosting infrastructure. Namely, risks of services downtime for their customers, including generative AI developers and end-users.¹⁷

Hyperscalers (large cloud service providers that can provide computing and storage services at enterprise scale) enjoy privileged access to limited hardware resources, enabling them to offer exclusive access to GPAI models developed internally, in collaboration with partners, or through investments, thereby creating serious barriers to entry.¹⁸ Amazon not only provides cloud infrastructure to Anthropic, Stability AI, and AI21, but also competes with them by offering its own GPAI models on its Amazon Bedrock platform.¹⁹ Cloud hosts have unparalleled power to monitor, detect, and stifle competitors emerging on their cloud platform²⁰, while resources generated using upstream dominance allows them to influence research and development downstream.²¹ 

Big Tech-backed research papers are frequently cited in research, indicating a notable uptake of their ideas among the wider scientific community.²² Their ownership and operation of software development frameworks – standardised processes for developing AI and other software, including vast data repositories, training methods, and evaluation tools – shapes AI development and deployment by ensuring that engineers adhere to development practices that are interoperable with Big Tech products.²³ Although PyTorch functions as a research foundation within the Linux Foundation, it is bankrolled by Meta. Google’s TensorFlow programming is specifically designed for Google’s Tensor Processing Units (TPUs), Google’s inhouse AI semiconductors, available on Google Cloud Platform, facilitating Google’s vertical integration from development practices to compute resources.  

Developers of the most advanced GPAI models currently on the market have a first-mover advantage. It is more straightforward for OpenAI to maintain and attract business users, because some of those clients may be hesitant to switch to GPAI competitor due to data security concerns or the cost and complexity of moving their data²⁴, as previously witnessed in the cloud space²⁵. Having made the large initial investment, OpenAI have a head start, learning from and building upon their already advanced model, while recovering that initial investment from the monetisation of GPT-4, as others seek to get their models off the ground.

Early entry allows these providers to purchase access to compute and data at rates that will be lower than for new entrants if increased demand pushes prices up.²⁶ It affords them greater time to improve their models’ performance through finetuning, build brand visibility, and a strong consumer base. They have a head start in harvesting user data to feed into future training runs and developing higher performance models. This reinforces their prominence, whereby greater performance attracts more users, builds better trust in the model at the expense of new and unknown alternatives, and gives them capital to continue crowding out the market.

The best models currently belong to Big Tech, not universities –in 2022, industry produced 32 leading models, while academia produced three.²⁷ This reduces academic access to cutting-edge models for evaluations of systemic risks and the development of effective mitigation measures. Compared to nonprofits and universities, the private sector has the most resources to recruit the best talent, use large amounts of compute, and access data, both in quantity and quality, all of which is required to build state-of-the-art GPAI. This limits the amount of high skilled workers, needed to build the most competitive AI to industry, hindering academia in training the next generation of advanced AI developers.²⁸ As a result, supply is not meeting demand, not least because there is a race to find better engineers, who can discover algorithmic innovations that reduce the amount of compute or data – and costs – required for training.²⁹ SMEs and startups must try to attract talent away from more resourceful incumbents, who can offer bigger employee remunerations.

GPAI models, and generative AI systems, involve fixed costs in development, such as pretraining and fine-tuning compute resources, data collation, and inhouse and outsourced labour, and relatively low marginal costs in deployment.³⁰ These economies of scale are a significant barrier to entry for startups, as they would need to develop and deploy models and systems at scale from the outset in order to compete.³¹ It is usually more realistic for smaller European providers to fine-tune American models into customised models or domain-specific systems that require less compute, data, and labour.³² But this still renders downstream developers and deployers dependent on larger upstream model providers.  

The general purpose nature of these AI models and systems allows for versatile and flexible deployment settings, which will increase their uptake throughout diverse industries. For providers, this allows them to spread substantial initial investment spending across these use cases, while their downstream customers will save money by deploying the same capability across different tasks.³³ These economies of scope are a challenging barrier to entry for Big Tech challengers, as they would need to be servicing the same range of sectors in order to compete.³⁴ 

3) What are the main drivers of competition (i.e., the elements that make a company a successful player) for the provision, distribution or integration of generative AI systems and/or components, including AI models?

The leading GPAI models and generative systems are more performant because they have access to the most or best data, computational resources, and skilled developers. These factors allow them to attract more users; amass more data and capital to purchase more chips; access more cloud infrastructure; develop better models and applications; and, in turn, attract more and better developers.³⁵ OpenAI engineers can make up to $800,000 per year, salaries that no SME or startup, especially in Europe, could afford.³⁶ Importantly, as their models become increasingly capable, doors open up for the leading GPAI providers to monetise and advertise their models, as well as enter into commercial relationships with downstream system developers, which not only provides even greater user-facing visibility, but can also offer access to specialised domain or task specific data that is held by particular downstream parties.³⁷ If they are unable to obtain such unique data from these partnerships, then their increased revenues can be used to purchase it elsewhere.

These network effects are accelerated by data feedback effects, whereby general purpose AI developers leverage data generated from the conversations between the system and its users to advance capabilities.³⁸ While data generated during user interactions is not automatically used to train the model, since developers need to vet feedback for quality and safety, this may change if innovations lead to safe automatic continuous learning post-deployment.³⁹ OpenAI recently announced that ChatGPT will be able to memorise conversations in order to better tailor its responses to user preferences.⁴⁰ The more GPAI developers can refine a model toward their customers, the more useful it will be for customers, who will be less inclined to try out another challenger platform.⁴¹

Even if they mainly use feedback data in aggregate to understand wider trends, this is still a considerable competitive advantage for the most widely used models and systems that can collect the most amount of user data, providing more enriched aggregate analysis. Companies like OpenAI are at a particular advantage because they are present at both the model and system level, allowing them to use system level feedback to improve their model. European GPAI system developers, who will be largely reliant on building their systems upon American GPAI models, would be unable to benefit from this feedback loop, because they would be unable to use the data generated from their system to improve the underlying model. Superior access to resources – capital, computing power, data, and expertise – enables the creation of superior models. These models attract more consumers, resulting in increased revenue. This revenue, in turn, provides access to even better resources, thus perpetuating the cycle of developing high-quality models, asserting market dominance, and the capacity to foreclose competition from challengers.

4) Which competition issues will likely emerge for the provision, distribution, or integration of generative AI systems and/or components, including AI models? Please indicate to which components they relate.

While user feedback may not necessarily be leveraged for marketing services or constructing advertising profiles, enhanced capabilities can improve downstream GPAI services. This enables more precise customisation to consumer preferences, thereby driving adoption rates and establishing user loyalty.⁴² End users and business users will be locked in unless it is sufficiently practical to port data when switching to an alternative. Even with adequate interoperability, they may be discouraged from trying alternatives due to the convenience of accessing all their GPAI and related tools, services, and plug-ins via the one established platform. Such lock-in creates a positive feedback loop for the GPAI provider, positioning the model for further monetisation, as it continues to progressively build a more robust and holistic picture of the user, thereby empowering it to offer more tailored targeting of products, including the provider’s other downstream GPAI services in adjacent markets, such as search, social media, app stores and productivity software. 

This grants the provider the power to engage in unfair and abusive practices. Dominance in both the GPAI model and system market coupled with dominance in these adjacent markets allows large incumbents to buttress their dominance in each by bundling their GPAI service with their other services in search, online platforms, or productivity software. Beyond the convenience of a one-stop shop interface, users may be unable to switch if doing so means they would lose access to another tied service. The first-mover advantage of the currently leading GPAI models – GPT-4, Gemini, Llama 2 – allows them to enjoy network effects, and with customer lock-in, switching costs will also act as a barrier to entry for SMEs and startups.

5) How will generative AI systems and/or components, including AI models likely be monetised, and which components will likely capture most of this monetization?

As recognised by Stanford researchers⁴³, when GPAI model providers grant sufficient access to their models to downstream system developers, through an application programming interface (API), they are operating AI as a platform, similar to platform as a service (PaaS) for software, allowing them to access models to adapt to specific user facing GPAI and AI systems, like an app store for app developers. Beyond this, OpenAI, for example, also allows plugin integrations that connect third-party apps to the paid version of ChatGPT (based on GPT-4, not GPT-3.5, as in the free version). This increases capabilities by allowing ChatGPT to retrieve real-time information, proprietary information, and action real-world user instructions.⁴⁴ Plugins empower ChatGPT to act as a platform by enabling it to select options among different providers or present different options to the user.⁴⁵

More recently, OpenAI launched GPT Store⁴⁶, so its non-expert paying users can find and build fine-tuned versions of the ChatGPT GPAI system.⁴⁷ All of this attracts third-party app and plugin developers to OpenAI’s ecosystem, rendering more applications compatible with its main GPAI system, while providing OpenAI with oversight on developments that threaten their offerings.⁴⁸ Smaller plugin providers, in particular, may come to rely on platforms like ChatGPT, the fastest growing consumer application in history⁴⁹, for their user base⁵⁰, but OpenAI may use this position to provide their own competing applications downstream.⁵¹ As OpenAI expands its plug-in offerings, their platform becomes more appealing for plug-in developers, allowing OpenAI to draw in more plug-ins, which increases the amount of consumers, motivates more developers, and makes their platform ever-more appealing.⁵² 

6) Do open-source generative AI systems and/or components, including AI models compete effectively with proprietary AI generative systems and/or components? Please elaborate on your answer.

The considerable costs required to develop general purpose AI models from square one and then deploy them at scale, apply equally to closed and open models. While open source licenses offer new entrants more accessibility at the model level (parameters, data, training support), open source models do not address compute concentration in the markets for semiconductors and cloud infrastructures.⁵³ All tahe frontier open source models rely on Big Tech compute⁵⁴: Meta’s Llama 2 runs on Microsoft Azure; UAE-based Technology Innovation Institute’s Falcon 180B model runs on AWS⁵⁵; and Mistral’s Mixtral models runs on Google Cloud⁵⁶. EleutherAI’s GPT-NeoX-20B runs on NVIDIA-backed, AI-focused CoreWeave Cloud⁵⁷, who rent out GPUs at an hourly rate⁵⁸, allowing them to scale from 3 to 14 data centres in 2023⁵⁹, but remains well below Meta and Microsoft, who are NVIDIA’s top GPU customers⁶⁰. Microsoft have committed to billions of dollars in investment in CoreWeave in the coming years to secure access to NVIDIA’s GPUs⁶¹ ahead of their real rivals, AWS and Google Cloud⁶².

At first glance, Meta’s Llama 2 meets the definition of a free and open source license in the recently agreed AI Act, considering that Meta publishes the model parameters, including weights, and information on model architecture and model usage. However, Meta does not publish information on the model training data – precisely why providers of such models are required to do so under the AI Act, regardless of whether they present systemic risks or not. Nevertheless, Meta’s Llama 2 licence⁶³ is not open source⁶⁴, as widely recognised⁶⁵, particularly by the Open Source Initiative⁶⁶, whose open source definition⁶⁷ is the global community benchmark. Meta does not allow developers to use Llama 2 or its outputs to improve any other large language model (LLM), and app developers with more than 700 million monthly active users must request a license from Meta, which Meta is not obliged to grant, presumably if it feels competitively challenged.⁶⁸ By permitting commercial use of Llama 2, on a small and non-threatening scale, Meta leverages unpaid labour to enhance the model’s architecture, enabling it to monetise such improvements, as endorsed by their CEO.⁶⁹

European SMEs and startups will still be highly dependent on the largest chips developers (largely NVIDIA) and cloud providers (Amazon, Microsoft, and Google), as well as the leading AI development frameworks (Meta and Google). This dependence affirms and asserts gatekeepers’ market monitoring powers that can anticipate and foreclose competition from innovative new entrants through self-preferencing or copying.⁷⁰ Even with leveraging open source GPAI models, EU players will still need major funding to train and deploy their GPAI models, if they are to be competitive, which will need to come from EU governments and venture capital firms, if they are not to be bought up by Big Tech. Otherwise, EU GPAI developers will be limited to fine-tuning existing models, open or closed, which does not empower downstream parties to fundamentally alter data and design choices that were shaped upstream.⁷¹ 

According to Mistral, their latest Mixtral 8x7B model matches or exceeds Meta’s Llama 2 70B and OpenAI’s GPT-3.5 on many performance metrics and is better on maths, code, and multilingual tasks, while using fewer parameters during inference.⁷² By utilising only a portion of the overall parameters per token, it effectively manages costs and reduces latency. It is open source (though this is reasonably disputed)⁷³, released under the Apache 2.0 license, and free for academic and commercial use. Until recently, the European developer’s capacity to develop competitive GPAI models was supported by €385 million, among other funding, including from American venture capital firms, such as Andreessen Horowitz and Lightspeed.⁷⁴ Building on their successes, and seeking to secure their long-term financing and operational viability, they have reached a deal with Microsoft, who will invest €15 million. This allows Mistral to use Microsoft supercomputers to train their GPAI models on Azure and access Microsoft customers for greater distribution of their products, while it allows Microsoft to offer Mistral models as premium features for its customers.⁷⁵ The partnership positions Microsoft with a leading presence in both the open source model market (through Mistral) and closed proprietary model market (through OpenAI).⁷⁶ While Microsoft’s investment in Mistral currently doesn’t confer ownership stake, it could convert to equity in Mistral’s subsequent funding round.⁷⁷ 

This episode vividly illustrates that when an open source alternative appears to threaten the most well-funded proprietary models, such as GPT-4, those funding the challenged model quickly move in to stake their financial interests in the upstart new entrant, foreclosing competition. Microsoft is hedging its bets in case Mistral’s models should come to outperform those of their other investment, OpenAI, in case open source AI becomes the dominant business model or ecosystem that generates the greatest economic value. While open source holds promise for greater transparency and accessibility, this development underscores that it is incredibly difficult for open source AI models to get off the ground without the financial backing of Big Tech.

It highlights that the AI Act threshold for classifying models as systemic – those models trained on compute using 10^25 or more FLOPS – should not be raised, as desired by industry. During trilogue discussions, and even now, the French government argue that the threshold should be 10^26, in part due to concerns that their national champion, Mistral, would reach the threshold within a year. The deal between Microsoft and Mistral makes it clear that reaching that threshold, which depends on vast resources in cloud computing capacity, requires funding from those with entrenched positions in digital markets.

The partnership has undermined the self-proclaimed⁷⁸ European independence of Mistral.⁷⁹ For EU policymaking, naturally there is concern about conflicts of interest during the AI Act negotiations, as highlighted by Green MEPs in a letter to the Commission⁸⁰, especially considering that this deal was likely also under negotiation over the same period. While this may not reach a threshold of a competition breach or market abuse, the Commission should be concerned when European AI startups, that are able to achieve a certain scale, can only survive through gatekeeper funding. This renders the European AI startup vulnerable to being co-opted as a useful voice or vehicle for Big Tech lobbying that seeks to minimise their compliance burden at the expense of safety for European consumers. For highly capable or impactful open model GPAI, risks are amplified by the inability of the original provider to effectively remediate or recall a dangerous open model after it has been released and downloaded innumerable times. While their inherent transparency may have benefits for accountability, it can also provide malicious actors with access to the model weights, enabling the discovery and exploitation of vulnerabilities or the circumvention of guardrails to generate harmful illegal outputs, including the development of lethal weapons, cyberattacks against critical infrastructure, and electoral manipulation.

7) What is the role of data and what are its relevant characteristics for the provision of generative AI systems and/or components, including AI models?

While publicly available data is still broadly accessible to new entrants, public data can be low quality, leading to less capable and even dangerous models.⁸¹ Stanford researchers found that one of the leading public datasets, LAION-5B, includes thousands of images of child sexual abuse material.⁸² Lensa, an image generator built on top of the GPAI model Stable Diffusion, which is trained on LAION-5B, was found to create realistic sexualised and nude avatars of women, particularly from traditionally marginalised communities, with less propensity to do the same in male renditions when prompted.⁸³

Proprietary datasets can offer more specialised and unique data that will give a model a deeper understanding of the world.⁸⁴ This not only makes a model more capable, but also allows it to be more easily aligned with our interests since it can understand us better in theory. This mitigates biases and inaccuracies within models, generating trust and encouraging adoption, thereby facilitating positive feedback loops for those with the best data. Big Tech’s accumulated data banks – both personal data from their B2C markets and non-personal data from their B2B/B2G markets – gives them an edge, as they have access to the public datasets that new entrants would, as well as their own enormous proprietary datasets which are closed off to new entrants.⁸⁵ High quality proprietary data is often held in downstream companies that specialise in a certain domain and have gathered data on that domain’s customers.⁸⁶ Google’s $2.1 billion acquisition of Fitbit gives them millions of users’ health data, which has been tracked for over a decade, as well as access to Fitbit’s insurance partners.⁸⁷ This allows Google to leverage this wealth of proprietary data if they seek to fit their GPAI models with health features tailored to their users, giving them a competitive edge over models without this capability. Such an acquisition is beyond the reach of European startups.

The innovation gap is widened further by Big Tech’s greater experience in developing models, housing the best expertise, and scraping, labelling, and analysing data. Moreover, search engine providers, namely Google and Microsoft⁸⁸, can leverage public data more effectively by using web indexes to filter out meaningless or useless information, leaving behind the high quality public data, which may be a more efficient process since the web data is more vast than proprietary datasets.⁸⁹ One way European SMEs and startups could catch up is through algorithmic innovations that can do more with less data, but this requires access to the best talent, which is another increase in costs. The current competitive frontier goes even further in that ChatGPT and Gemini will compete on how many other systems they are connected to, providing them with continuous real-time up-to-date data.

Successes in commercial GPAI and its vast potential across innumerable use cases have also led to data providers seeking to cash in on the AI gold rush by monetising their data for AI training.⁹⁰ When data was free and easy to access, the current GPAI model leaders got in early.⁹¹ As content creators, or the platforms on which such content is hosted, restrict access, or seek remuneration, new entrants may face barriers to entry with prohibitively costly data on top of exorbitant compute costs. If legislation and judicial rulings reassert rightsholders’ intellectual property rights, public data could become increasingly scarce, pushing the price up further.⁹² European SMEs and startups could turn to synthetic data as an alternative to proprietary data, but more resources on compute are needed to generate such artificial information.⁹³ Saving on data pushes costs up elsewhere. Using AI models to generate data for future models can transfer errors and bugs from the old model to the new one.⁹⁴ 

8) What is the role of interoperability in the provision of generative AI systems and/or components, including AI models? Is the lack of interoperability between components a risk to effective competition?

The concentrated cloud market, upon which GPAI is developed and deployed, combined with the lack of interoperability between AWS, Azure, and Google Cloud Platform, provides single points of failure that could be disruptive and destabilising across sectors, given the market share of these three providers.⁹⁵ As single failure points, they are an attractive target for cyberattacks by malicious actors.⁹⁶ If such an attack were successful, it would cut off not only the cloud infrastructure platform, but also the general purpose AI model and the many downstream generative AI systems deriving from it that run on the same cloud. The lack of interoperability means critical applications, such as those in defence, health, or finance, cannot be easily migrated to another cloud provider in order to get them up and running again.⁹⁷ In a scenario where a hostile nation or well-funded terrorist group penetrates a single point of failure to cripple critical services, a full blown assault on both private and public databases could not only cause widespread disruption, but it may also be difficult to detect, making it all the more challenging to restore organisational data to a safe and reliable state.⁹⁸ Concentrations at the model level can also produce similar security risks given that any vulnerability in a model upon which user-facing applications are built could produce systemic hazards, exacerbated by emergent capabilities that can develop unpredictably with further fine-tuning at the system level.⁹⁹

9) Do the vertically integrated companies, which provide several components along the value chain of generative AI systems (including user facing applications and plug-ins), enjoy an advantage compared to other companies? Please elaborate on your answer.

General purpose AI models and generative AI systems that are plugged into third-party applications can operate as platforms, or they can be vertically integrated into another platform.¹⁰⁰ To catch up with Google, Microsoft integrated GPT-4 as Bing Chat into its platforms, Bing search and Edge¹⁰¹, and as Copilot into its 365 productivity suite.¹⁰² As a response, Google is testing its integration of generative AI into Google Search in the US¹⁰³ – Search Generative Experience (SGE)¹⁰⁴ – which allows Google to leverage an adjacent market, bolstering its advertising revenues and strengthening its grip on online website traffic.¹⁰⁵ This illustrates a transition from search engines to answer engines.¹⁰⁶ Users may be less inclined to visit third-party websites, provided as citations or footnotes, since the answer is already in the chatbot interface, to which advertisers turn their attention at the expense of third-party websites.¹⁰⁷ This could allow Google’s generative search engine to benefit from the intellectual property of others, whose data is fed into the generative interface, not only without compensation, but also without the usual user traffic to their site.

For users, and society at large, reliance on generative search engines risks reducing the accuracy of information, as it is difficult to distinguish between outputs derived from the training data and those from the search results, and the hidden hallucinations therein.¹⁰⁸ Stanford found that users’ perception of utility is inversely correlated with the precision of citations purporting to support claims made by the generative search/answer engine.¹⁰⁹ While Bing Chat achieves the highest citation precision rate, users find it the least useful, whereas YouChat has the lowest citation precision rate, but users deem it the most useful. Given that upstream GPAI models are likely to significantly augment the user experience or perceived utility, if not accuracy, of downstream search or answer engines, users will be increasingly drawn to these platforms¹¹⁰, which will be a barrier for entry for GPAI model providers that don’t compete on both nodes of the value chain by only offering GPAI models, but not GPAI-infused search engines.¹¹¹ 

Google is present throughout the entire AI value chain¹¹²: it produces its own semiconductors (TPUs), hosts its own cloud infrastructure (Google Cloud), develops its own GPAI models (PaLM-2 and Gemini), provides GPAI systems (Gemini, so users can interact directly with the model)¹¹³ and integrates those systems into its platforms (Search Generative Experience). From these platforms, it also gathers data that can be used to improve future iterations of Gemini, increasing the model’s capabilities and utility. The revenues can also be used to purchase more compute, data, and talent. Vertically integrated corporations will have easier access to unique data, such as conversations between users on their platforms.¹¹⁴ 

Vertical integration risks institutionalising unbreakable tech oligopolies, hindering innovative efforts of European SMEs and startups, weakening consumer choice, and inflating the cost of gatekeeper services beyond their value, either in subscription charges or data extraction. While NVIDIA is currently leading on GPUs, Microsoft, Google and Meta are all seeking to compete by developing their own chips for AI.¹¹⁵ If Microsoft or Google were to overtake NVIDIA, their vertical integration, either legally or in practice, from semiconductors (Microsoft’s ; Google’s TPUs) to cloud (AWS; GCP) to models (GPT-4; Gemini) to systems (Bing, ChatGPT, Copilot; Gemini) could ensure that AI development becomes a two-horse race, as it would be incredibly difficult, if not impossible, for challengers to compete at each level of that value chain. In this scenario, Microsoft or Google could then engage in unfair and abusive practices, such as limiting the access of GPAI model and system competitors to key ingredients like chips and cloud infrastructure.

Microsoft’s CEO, Satya Nadella, claims¹¹⁶ that his company’s partnership with OpenAI challenges vertically integrated companies like Google.¹¹⁷ Yet, concerns are mounting that the partnership may amount to a decisive influence under Article 3 of the Mergers Control Regulation, given their exclusivity arrangements, as well as the successful pressure Microsoft put on OpenAI’s board to reinstate their fired CEO, Sam Altman, including offering him and other OpenAI staff employment. ¹¹⁸ This raises questions about OpenAI’s ability to operate independently and be considered a separate company that is not vertically integrated with Microsoft “in spirit”, if not in narrow legal terms. The grey-area manoeuvrings of Altman’s firing and rehiring illuminate that Microsoft can control OpenAI, without acquiring it or breaking antitrust or merger laws, by leveraging its exclusive access to their leading GPAI models and the scaleups access to gatekeeper’s compute – an arrangement that prohibits OpenAI from migrating their models to other cloud providers .¹¹⁹ 

10) What is the rationale of the investments and/or acquisitions of large companies in small providers of generative AI systems and/or components, including AI models? How will they affect competition?

Cloud providers typically prefer to create partnerships with established GPAI providers, affording the latter preferential access to scare computational resources and investment opportunities.¹²⁰ This is cheaper for the GPAI developer than paying access via on-demand rates or via upfront or subscription charges, let alone building their own data centre. OpenAI must use Azure, while Microsoft can integrate OpenAI products across all its offerings¹²¹, with priority access.¹²² 

11) Do you expect the emergence of generative AI systems and/or components, including AI models to trigger the need to adapt EU legal antitrust concepts?

While the Digital Markets Act (DMA) is not strictly an antitrust instrument, it does seek to ensure open digital markets and to provide an additional lever in the Commission toolkit for lengthening antitrust investigations. Although the DMA does not explicitly cover AI, generative AI should be in-scope when it is integrated into a core platform service.¹²³ 

At the infrastructural level in GPAI model and system development and deployment, cloud computing is already listed as a core platform service. However, none have been designated at the time of writing, primarily due to hyperscalers not meeting the quantitative thresholds given that they don’t technically have end-users, according to the DMA definition.¹²⁴ There is recognition that business users may also be counted as end users when they use cloud computing services for their own purposes (Recital 14 of the DMA). This should be included when counting active end users of cloud computing services, given that AI labs such as OpenAI and Anthropic (and the many other businesses fine-tuning their GPAI models via an API that is run on cloud services) might all be considered end-users¹²⁵ of Azure, Google Cloud Platform and AWS, rather than business users¹²⁶, based on DMA definitions. This could mean that the cloud services of hyperscalers would be designated as core platform services, and would thereby ensure that the oligopolist cloud market is tackled by EU ex-ante regulation, rather than complaints brought by cloud service challengers that would struggle to afford lengthy legal proceedings.

As in the Commission’s initial DMA impact assessment¹²⁷, the end user and business user definitions should equally cover infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) in order to avoid loopholes in the DMA’s application. If this is not already the case, the Commission should amend and update the methodology and list of indicators in the Annex of the DMA through delegated acts (Article 3(7)) to ensure the DMA can mitigate further concentrations in the cloud market, which underpins GPAI development and deployment. To ensure the DMA remains fit for purpose given the rapid advances in AI, as well as API as an intermediation platform, the Commission should consider whether they have the legal basis to update the list of core platform services to accommodate GPAI models and systems.¹²⁸ 

According to reports, Microsoft has cautioned competing search engines that it will terminate licenses providing access to its Bing search index if they continue to use it for generative AI chat development.¹²⁹ Article 6(2) of the DMA prohibits gatekeepers from using non-publicly available data, generated by business users or their customers using the core platform service, to compete with those business users. This could help to ensure that GPAI model providers are hindered in preventing GPAI system developers, dependent on the model provider for API access, from competing using data generated through their use of the cloud service.¹³⁰ Although Bing has not been designated, it may reach the threshold if its integration with OpenAI models makes it more attractive to end users and business users.

Given their foundational and systemic function across the economy and society, large cloud computing and GPAI model providers should be regulated like public utilities, adhering to similar rules on non-discrimination, equitable treatment of all customers, and assurance of safety and security.¹³¹ Since antitrust law primarily seeks to address monopolies, public utility framing is critical, as the oligopoly in the cloud market may make the AI market more concentrated in the coming years.¹³² 

The Commission could also consider the feasibility of structural separation to prohibit companies from owning both GPAI models and other technologies and platforms that enable them to engage in unfair and abusive practices.¹³³ While this could be achieved through antitrust law, it typically requires a lengthy investigation process, which means that regulation may be more viable. As in the AI Act, at present, the amount of compute used during training is one of the best ways of quantifying a GPAI model’s impact and capabilities. Based on the current state of the art, the Commission could use compute as a proxy for determining the largest market players in order to apply structural remedies that would mitigate market failures.¹³⁴

12) Do you expect the emergence of generative AI systems to trigger the need to adapt EU antitrust investigation tools and practices?

Notwithstanding the already increased scrutiny of competition authorities towards issues related to the digital economy in recent years¹³⁵, detecting and assessing potential competition law infringements will become increasingly complex. Such complexity is particularly pronounced when facing companies with business models that deploy network effects or benefit from ecosystems¹³⁶, which generate and collect data to enhance value. This data allows companies to refine algorithms and services, which subsequently increases their value on markets. Their business models often use GPAI models and systems, blockchain, IoT, robotics, algorithms, and machine learning¹³⁷ to offer services, such as providing search results (Google), recommending products (Amazon) or accommodation (Airbnb).¹³⁸ These digital platforms centered around data are changing competitive dynamics rapidly, posing considerable challenges for competition authorities.

As a result, the current competition law enforcement framework and tools will be under pressure. It might be necessary to account for increasingly more diverse parameters, beyond the traditional focus on output and prices. For example, in fast-moving and dynamic markets powered by AI, competition authorities will be required to capture and understand data more quickly. In addition, in the context of devising a market definition, which has become more complex for digital platforms, the traditional SSNIP test may no longer suffice. Similarly, while the EU Merger Regulation can be somewhat adapted, it doesn’t adequately capture collaborations where companies like Microsoft partner with, and provide minority investments, in other parties (such as OpenAI) gaining influence and control without outright ownership.¹³⁹ If it is not possible to tackle these kinds of relationships, of vertical outsourcing rather than vertical integration, then reassessment and revision of the Merger Regulation is needed.

GPAI also enables companies to engage in new kinds of anticompetitive behaviour (see also the answer to question 4). For example, algorithms enable companies to automatically monitor the prices of competitors in real time and then re-price (algorithmic collusion). Companies with substantial market power in a certain market, may use GPAI to reinforce their market power in another market or to exclude competitors (as seen in the Google Shopping Case¹⁴⁰).

In view of the transformations and advancements stemming from the emergence and deployment of GPAI, there is a significant risk that competition authorities may struggle to grasp companies’ behaviour and market dynamics in a timely manner in order to prevent anti-competitive effects from taking place. Considering that the European Commission directly enforces EU competition rules and possesses an extensive toolkit for antitrust investigations, it is imperative to bolster enforcement tools and reevaluate how competition is analyzed to ensure EU competition policy remains future proof. By fostering a competitive GPAI market and value chain, other regulations – such as the AI Act, the Product Liability Directive, the forthcoming AI Liability Directive, the Data Act, the GDPR etc. – become more enforceable. Monopolists and oligopolists should not become too big to regulate, treating fines for non-compliance with these EU laws as operating expenses.¹⁴¹ Better compliance improves AI safety, fostering trust, and accelerating adoption of beneficial AI across the EU, while levelling the playing field for innovative European AI startups to offer competitive alternatives.

---

^{↩ 1} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 2} UK CMA. “AI Foundation Models Initial Report”. 

^{↩ 3} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.  

^{↩ 4} Malik. “OpenAI’s ChatGPT now has 100 million weekly active users”.

^{↩ 5} Stringer, Wiggers, and Corrall. “ChatGPT: Everything you need to know about the AI-powered chatbot”. 

^{↩ 6} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 7} Carugati. “The Generative AI Challenges for Competition Authorities”.

^{↩ 8} Techo Vedas. “NVIDIA has 90% of the AI GPU Market Share; 1.5 to 2 million AI GPUs to be sold by NVIDIA in 2024”. 

^{↩ 9} Statista. “Semiconductor foundries revenue share worldwide from 2019 to 2023, by quarter”.

^{↩ 10} Whittaker, Widder, and West. “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI”.

^{↩ 11} Field. “OpenAI CEO Sam Altman seeks as much as $7 trillion for new AI chip project: Report”. 

^{↩ 12} Informed by discussion with Friso Bostoen, Assistant Professor of Competition Law and Digital Regulation at Tilburg University.

^{↩ 13} AI Now Institute. “Computational Power and AI”. 

^{↩ 14} Statista. “Amazon Maintains Cloud Lead as Microsoft Edges Closer”.

^{↩ 15} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 16} Belfield and Hua. “Compute and Antitrust”.

^{↩ 17} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 18} Bornstein, Appenzeller and Casado. “Who Owns the Generative AI Platform?”

^{↩ 19} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 20} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 21} Kak, Myers West, and Whittaker. “Make no mistake – AI is owned by Big Tech”. 

^{↩ 22} Giziński et al. “Big Tech influence over AI research revisited: memetic analysis of attribution of ideas to affiliation.”

^{↩ 23} Whittaker, Widder, and West. “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI”.

^{↩ 24} Economist. “Could OpenAI be the next tech giant?”.

^{↩ 25} Savanta. “European cloud customers affected by restrictive licensing terms for existing on-premise software, new research finds”.

^{↩ 26} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 27} Standford University HAI. “AI Index Report 2023”. 

^{↩ 28} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 29} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 30} Ada Lovelace Institute. “Foundation models in the public sector”.

^{↩ 31} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 32} Leicht. “The Economic Case for Foundation Model Regulation”.

^{↩ 33} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 34} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 35} Hausfeld. “ChatGPT, Bard & Co.: an introduction to AI for competition and regulatory lawyers”.

^{↩ 36} Constantz. “OpenAI Engineers Earning $800,000 a Year Turn Rare Skillset Into Leverage”.

^{↩ 37} Schrepel and Pentland. “Competition between AI Foundation Models: Dynamics and Policy Recommendations”.

^{↩ 38} OpenAI. “How your data is used to improve model performance”.

^{↩ 39} UK CMA. “AI Foundational Models Initial Report”. 

^{↩ 40} OpenAI. “Memory and new controls for ChatGPT”.

^{↩ 41} UK CMA. “AI Foundational Models Initial Report”.

^{↩ 42} Ibid.

^{↩ 43} Stanford University HAI. “AI Accountability Policy Request for Comment”.

^{↩ 44} OpenAI. “Chat Plugins”. 

^{↩ 45} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 46} OpenAI. “Introducing the GPT Store”.

^{↩ 47} Sentance. “The GPT Store isn’t ChatGPT’s ‘app store’ – but it’s still significant for marketers”.

^{↩ 48} OpenAI. “ChatGPT plugins”.

^{↩ 49} Reuters. “ChatGPT sets record for fastest-growing user base – analyst note”.

^{↩ 50} UK CMA. “AI Foundational Models Initial Report”.

^{↩ 51} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 52} UK CMA. “AI Foundational Models Initial Report”.

^{↩ 53} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 54} Whittaker, Widder, and West. “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI”.

^{↩ 55} Jackson. “TII trains state-of-the-art LLM, Falcon 40B, on AWS model”.

^{↩ 56} Reuters. “Google Cloud partners with Mistral AI on generative language models”.

^{↩ 57} Hjelm. “Looking Ahead to 2023: How CoreWeave Is Using NVIDIA GPUs to Advance the New Era of AI and Machine Learning”.

^{↩ 58} Krazit. “How CoreWeave went all-in on Nvidia to take on Big Cloud”.

^{↩ 59} Economist. “Data centres improved greatly in energy efficiency as they grew massively larger”.

^{↩ 60} Elder. “Sell Nvidia”. 

^{↩ 61} Novet. “Microsoft signs deal for A.I. computing power with Nvidia-backed CoreWeave that could be worth billions”.

^{↩ 62} Haranas. “Microsoft’s CoreWeave Deal ‘Adds AI Pressure’ To AWS, Google”.

^{↩ 63} Meta. “Request access to Llama”.

^{↩ 64} OpenUK. “State of Open: The UK in 2024 Phase One AI and Open Innovation”.

^{↩ 65} Tarkowski. “The Mirage of Open-source AI: Analysing Meta’S LLaMa 2 release strategy”.

^{↩ 66} Open Source Initiative. “Meta’s LLaMa 2 license is not Open Source”.

^{↩ 67} Open Source Initiative. “The Open Source Definition”.

^{↩ 68} OpenSource Connections. “Is Llama 2 open source? No – and perhaps we need a new definition of open…”.

^{↩ 69} Whittaker, Widder, and West. “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI”.

^{↩ 70} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 71} Whittaker, Widder, and West. “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI”.

^{↩ 72} Jiang et al. “Mixtral of Experts”. 

^{↩ 73} Robertson. “France’s Mistral takes a victory lap”. 

^{↩ 74} Volpicelli. “Microsoft’s AI deal with France’s Mistral faces EU scrutiny”.

^{↩ 75} Volpicelli. “European lawmakers question Commission on Microsoft-Mistral AI deal”.

^{↩ 76} Murgia. “Microsoft strikes deal with Mistral in push beyond OpenAI”

^{↩ 77} Coulter and Yun Chee. “Microsoft’s deal with Mistral AI faces EU scrutiny”.

^{↩ 78} Mensch. X (Twitter) post.

^{↩ 79} Zenner. “Microsoft-Mistral partnership and the EU AI Act”

^{↩ 80} Volpicelli. “European lawmakers question Commission on Microsoft-Mistral AI deal”.

^{↩ 81} UK CMA. “AI Foundation Models Initial Report”. 

^{↩ 82} Ropek. “An Influential AI Dataset Contains Thousands of Suspected Child Sexual Abuse Images”.

^{↩ 83} Heikkilä. “The viral AI avatar app Lensa undressed me without my consent.” 

^{↩ 84} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 85} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 86} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 87} Austin. “The Real Reason Google Is Buying Fitbit”.

^{↩ 88} Hausfeld. “ChatGPT, Bard & Co.: an introduction to AI for competition and regulatory lawyers”.

^{↩ 89} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 90} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 91} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 92} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 93} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 94} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 95} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 96} Lutkevich. “Foundation models explained: Everything you need to know”.

^{↩ 97} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”. 

^{↩ 98} World Economic Forum. “Understanding Systemic Cyber Risk”. 

^{↩ 99} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 100} Ibid.

^{↩ 101} Techsyn. “Microsoft Integrates OpenAI’s GPT-4 Model Into Bing For A Powerful Search Experience”.

^{↩ 102} Sullivan. “Inside Microsoft’s sprint to integrate OpenAI’s GPT-4 into its 365 app suite”.

^{↩ 103} Google. “Supercharging Search with generative AI”.

^{↩ 104} Google. “Search Labs”. 

^{↩ 105} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 106} Carugati. “Competition in Generative AI Foundation Models”.

^{↩ 107} Carugati. “The Generative AI Challenges for Competition Authorities”.

^{↩ 108} Miller. “Generative Search Engines: Beware the Facade of Trustworthiness”.

^{↩ 109} Liu, Zhang, and Liang. “Evaluating Verifiability in Generative Search Engines”.

^{↩ 110} Vipra and Korinek. “Market concentration implications of foundation models: The invisible hand of ChatGPT”.

^{↩ 111} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 112} Narechania and Sitaraman. “An Antimonopoly Approach to Governing AI”.

^{↩ 113} Google. “Bard becomes Gemini: Try Ultra 1.0 and a new mobile app today”.

^{↩ 114} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 115} David. “Chip race: Microsoft, Meta, Google, and Nvidia battle it out for AI chip supremacy”.v

^{↩ 116} Hartmann. “Microsoft CEO defends OpenAI’s ‘partnership’ amid EU, UK regulators’ scrutiny”.

^{↩ 117} Smith. “Microsoft’s AI Access Principles: Our commitments to promote innovation and competition in the new AI economy”.

^{↩ 118} Irish Council for Civil Liberties et al. “Submission to European Commission on Microsoft-OpenAI “partnership” merger inquiry”.

^{↩ 119} Callaci. “The Antitrust Lessons of the OpenAI Saga”.

^{↩ 120} UK CMA. “AI Foundation Models Initial Report”.

^{↩ 121} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 122} Irish Council for Civil Liberties et al. “Submission to European Commission on Microsoft-OpenAI “partnership” merger inquiry”.

^{↩ 123} Informed by discussion with Friso Bostoen, Assistant Professor of Competition Law and Digital Regulation at Tilburg University.

^{↩ 124} Abecasis et al. “6 reflections on the recent designation of gatekeepers under the DMA”.

^{↩ 125} Digital Markets Act definition of active end users for cloud computing: “Number of unique end users who engaged with any cloud computing services from the relevant provider of cloud computing services at least once in the month, in return for any type of remuneration, regardless of whether this remuneration occurs in the same month.”

^{↩ 126} Digital Markets Act definition of active business users for cloud computing: “Number of unique business users who provided any cloud computing services hosted in the cloud infrastructure of the relevant provider of cloud computing services during the year. “

^{↩ 127} European Commission. “Impact assessment of the Digital Markets Act 1/2”: “Cloud services . . . provide infrastructure to support and enable functionality in services offered by others and at the same time offer a range of products and services across multiple sectors, and mediate many areas of society. . . They benefit from strong economies of scale (associated to a high fixed cost and minimal marginal costs) and high switching costs (associated to the integration of business users in the cloud). The vertical integration of the large cloud services providers and the business model they deploy has contributed to further concentration on the market, where it is very difficult for other less-integrated players, or market actors operating in just one market segment to compete. Consequently, these startups are likely to be completely reliant on large online platform companies.”

^{↩ 128} von Thun. “EU does not need to wait for the AI Act to act”.

^{↩ 129} Dixit. “Microsoft reportedly threatens to cut-off Bing search data access to rival AI chat products”.

^{↩ 130} Yasar et al. “AI and the EU Digital Markets Act: Addressing the Risks of Bigness in Generative AI”.

^{↩ 131} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 132} Informed by discussion with Friso Bostoen, Assistant Professor of Competition Law and Digital Regulation at Tilburg University.

^{↩ 133} von Thun. “After Years of Leading the Charge Against Big Tech Dominance, is the EU Falling Behind?”

^{↩ 134} Belfield and Hua. “Compute and Antitrust”. 

^{↩ 135}  Google Android decision; Apple Pay Investigation; Apple App Store investigation; Amazon’s use of marketplace seller data Investigation.

^{↩ 136} Lianos: Hellenic Competition Commission and BRICS Competition Law and Policy Centre. “Computational Competition Law and Economics: An Inception Report”..

^{↩ 137} Schrepel. “Collusion by Blockchain and Smart Contracts”.

^{↩ 138} Iansiti and Lakhani. “From Disruption to Collision: The New Competitive Dynamics”. 

^{↩ 139} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

^{↩ 140} T-612/17 – Google and Alphabet v Commission (Google Shopping)

^{↩ 141} Lynn, von Thun, and Montoya. “AI in the Public Interest: Confronting the Monopoly Threat”.

The Future of Life Institute (FLI) is an independent nonprofit organization with the goal of reducing large-scale risks and steering transformative technologies to benefit humanity, with a particular focus on artificial intelligence. Since its founding ten years ago, FLI has taken a leading role in advancing key disciplines such as AI governance, AI safety, and trustworthy and responsible AI, and is widely considered to be among the first civil society actors focused on these issues. FLI was responsible for convening the first major conference on AI safety in Puerto Rico in 2015, and for publishing the Asilomar AI principles, one of the earliest and most influential frameworks for the governance of artificial intelligence, in 2017. FLI is the UN Secretary General’s designated civil society organization for recommendations on the governance of AI and has played a central role in deliberations regarding the EU AI Act’s treatment of risks from AI. FLI has also worked actively within the United States on legislation and executive directives concerning AI. Members of our team have contributed extensive feedback to the development of the NIST AI Risk Management Framework, testified at Senate AI Insight Forums, participated in the UK AI Summit, and connected leading experts in the policy and technical domains to policymakers across the US government.

---

Europe must lead the way on innovating trustworthy AI

Policy recommendations for the next EU mandate

The rapid evolution of technology, particularly in artificial intelligence (AI), plays a pivotal role in shaping today’s Europe.

As AI capabilities continue to advance at an accelerated pace, the imperative to address the associated dangers becomes increasingly urgent. Europe’s future security is intricately linked to the formulation and implementation of measures that effectively mitigate the risks posed by AI technologies.

Myopic policies which fail to anticipate the possibly catastrophic risks posed by AI must be replaced with strategies that effectively combat emergent risks. Europe must continue leading the way on AI governance, as it has repeatedly shown that its digital policies create global ripple effects. Europe must seize this opportunity to ensure deployment of AI aligns with ethical considerations and prioritises the safety of individuals and societies.

---

The AI Act is a done deal. Now it’s time to implement it.

With the historic adoption of the AI Act, the world’s first comprehensive hard-law regulation of AI, the focus will shift to its effective implementation and enforcement. This also necessitates renewed attention to complementary legislation, particularly the AI Liability Directive (AILD), to establish a holistic regulatory framework and solidify the EU’s position as a global leader. Prioritising the following areas will ensure that the shared goal of trustworthy, innovative, and safe AI is achieved:

i. Ensure that the AI Office is robust and has the ability to perform the tasks it has been set.

To ensure the robustness and efficacy of the AI Office within the European Commission, a series of strategic recommendations should be implemented. Firstly, offering competitive salaries to attract and retain top talent is essential. Adequate remuneration not only motivates technical experts who are usually captured by industry but also reflects the value placed on their expertise. Moreover, appointing leaders who possess a deep understanding of AI technologies and the risks they pose is crucial in order to articulate the mission and objectives of the AI Office, garnering support and engagement from stakeholders within and outside the Commission.

Additionally, facilitating secondments from industry and civil society organisations, as the UK AI Safety Institute has done, can bring diverse perspectives and experiences to the AI Office, within the context of limited resources. Temporary exchanges of personnel allow for knowledge transfer and collaboration, enriching the office’s monitoring and enforcement capabilities.

Furthermore, seamless collaboration between governance and technical teams, supported by effective leadership, operations, and human resources management, is paramount. Mirroring the range of roles and salaries made available by entities like the UK AI Safety Institute, the AI Office must provide sufficient incentives to attract experts who will further the Office’s goals, as prescribed by the AI Act.

ii. Reboot the AI Liability Directive to safeguard against unchecked risks and ensure accountability.

As the EU moves past the elections, it’s necessary to resume work on the AI Liability Directive (AILD). The explosive growth of AI across manufacturing, healthcare, finance, agriculture and beyond demands a robust legal framework that provides victims with recourse for damages caused by AI, and thereby incentivises responsible development and deployment. Current Union fragmentation, resulting from disparate AI liability regimes, leaves citizens vulnerable under less protective liability approaches at the national level. It also leads to legal uncertainty that hinders European competitiveness and inhibits start-ups from scaling up across national markets.

The AILD would enable customers, both businesses and citizens, to understand which AI providers are reliable, creating an environment of trust that facilitates uptake. By establishing clear rules for different risk profiles – from strict liability for systemic GPAI models to fault-based liability for others – we can foster fairness and accountability within the AI ecosystem. As these frontier GPAI systems have the most advanced capabilities, they present a diverse range of potential and sometimes unpredictable harms, leading to informational asymmetries which disempower potential claimants. Moreover, the necessary level of care and the acceptable level of risk may be too difficult for the judiciary to determine in view of how rapidly the most capable GPAI systems are evolving.

Re-engaging with the Directive reaffirms the EU’s position as a global leader in AI regulation, complementing the AI Act and PLD to create a holistic governance framework. The implementation of harmonised compensatory measures, covering both immaterial and societal damages, ensures uniform protection for victims throughout the EU. By addressing liability comprehensively and fairly, the AI Liability Directive can unlock the immense potential of AI for good while mitigating its risks. This is not just about regulating technology, but about shaping a future where AI empowers humanity, guided by principles of responsibility, trust, and the protection of individuals and society.

See FLI’s position paper on the proposed AI Liability Directive here.

iii. Actively involve civil society organisations in the drafting of Codes of Practice.

It is essential for the Commission to actively involve civil society groups in the formulation of Codes of Practice, as sanctioned by Article 52e(3) and Recital 60s of the AI Act. Both are ambivalent about civil society’s role, stating that civil society “may support the process” with the AI Office, which can consult civil society “where appropriate”. Collaborating with civil society organisations on the drafting of Codes of Practice is crucial to ensure that the guidelines reflect the state of the art and consider a diverse array of perspectives. More importantly, Codes of Practice will be relied upon up to the point that standards are developed, a process which is itself far from being concluded. It is therefore crucial that the Codes of Practice accurately reflect the neutral spirit of the AI Act and are not co-opted by industry in an effort to reduce their duties under the AI Act.

Civil society groups also often possess valuable expertise and insights, representing the interests of the wider public and offering unique viewpoints on the technical, economic, and social dimensions of various provisions. Inclusion of these stakeholders not only enhances the comprehensiveness and credibility of the Codes of Practice, but also fosters a more inclusive and democratic decision-making process. By tapping into the wealth of knowledge within civil society, the Commission can create a regulatory framework that is not only technically robust but also aligned with European values, reinforcing the commitment to responsible and accountable AI development within the EU.

iv. Issue clear, concise, and implementable AI Act guidance.

Another key goal for the new Commission and AI Office is to commit to issuing timely, concise, and implementable guidance on AI Act obligations. Drawing from lessons learned with the implementation of past Regulations, such as the GDPR, where extensive guidance documents became cumbersome and challenging even for experts, the focus should be on creating guidance that is clear, accessible, and practical.

Article 3 section (2)(c) from the Commission’s Decision on the AI Office highlights its role in assisting the Commission in preparing guidance for the practical implementation of forthcoming regulations. This collaboration should prioritise the development of streamlined guidance that demystifies the complexities of specific duties, especially with regards to general-purpose AI (GPAI) models with systemic risk. The availability of clear guidance removes ambiguities in the text which can otherwise be exploited. It also makes duties for providers, such as high-risk AI system developers, comprehensible, especially for SME developers with limited access to legal advice. The Commission should view guidance as an opportunity to start building lines of communication with SMEs, including start-ups and deployers.

For example, Article 62 of the AI Act centres around serious incident reporting and calls on the Commission to issue guidance on reporting such incidents. The effectiveness of Article 62, in many ways, rides on the comprehensiveness of the guidance the Commission provides.

v. Proactively foster international collaboration.

As the new Commission assumes its role, it is critical that it empowers the AI Office to spearhead international collaboration on AI safety. In accordance with Article 7 of the Commission Decision establishing the AI Office, which highlights its role in “advocating the responsible stewardship of AI and promoting the Union approach to trustworthy AI”, it is essential for the Commission to ensure that the AI Office takes a leadership position in fostering global partnerships. The upcoming AI safety summit in South Korea in May 2024 and the subsequent one in France in 2025 present opportune platforms for the EU to actively engage with other jurisdictions. When third countries take legislative inspiration from the EU, the AI Office can steer international governance according to the principles it has established through the AI Act.

Given the cross-border nature of AI, and for the purpose of establishing legal certainty for businesses, the AI Office should strive to work closely with foreign AI safety agencies, such as the recently established AI Safety Institutes in the US, UK, and Japan respectively. Additionally, it must play a pivotal role in the implementation of global agreements on AI rules. In doing so, the EU can position itself as a driving force in shaping international standards for AI safety, reinforcing the Union’s commitment to responsible innovation on the global stage.

vi. Build relationships with national competent authorities and ensure seamless collaboration on enforcement.

In line with Article 59 of the AI Act, we urge the new Commission to closely monitor the designation of national competent authorities and foster a collaborative relationship with them for robust enforcement of the AI Act. The Commission should exert political capital to nudge Member States to abide by the 12-month timeline for designating notifying and market surveillance authorities by each Member State. While these national competent authorities will operate independently, the Office should maintain a publicly accessible list of single points of contact and begin building roads for collaboration.

To ensure effective enforcement of the AI Act’s pivotal provisions, Member States must equip their national competent authorities with adequate technical, financial, and human resources, especially personnel with expertise in AI technologies, data protection, cybersecurity, and legal requirements. Given the uneven distribution of resources across Member States, it is to be expected that certain Member States may require more guidance and support from the Commission and AI Office specifically. It is crucial that the AI Board uses its powers in facilitating the exchange of experiences among national competent authorities, to effectively ensure that differences in competencies and resource availability would not impede incident monitoring.

vii. Secure the future of AI regulation by addressing the AI Office funding challenge.

Establishing the AI Office as mandated by the AI Act is crucial for effective governance and enforcement. However, concerns arise regarding the proposed funding through reallocation from the Digital Europe Program, originally geared towards cybersecurity and supercomputing. This approach risks diverting resources from existing priorities while potentially falling short of the AI Office’s needs. Moreover, the absence of dedicated funding within the current MFF (2021-2027) further necessitates a proactive solution.

While the new governance and enforcement structure presents uncertainties in cost prediction, established authorities like the European Data Protection Supervisor (EDPS) offer valuable benchmarks. Only in 2024, the EDPS has a budget of €24.33 million and employs 89 staff members. Another relevant benchmark is the European Medicines Agency (EMA), with 897 employees and a 2024 budget of €478.5 million (out of which €34.8 million is from EU budget). The AI Office would require comparable financial resources to other EU agencies, as well as an additional budget stream for compute resources which are needed to evaluate powerful models. Recent reports suggest a budget of €12.76 million once the AI Office is fully developed in 2025, an amount that will fall short of securing the proper governance and enforcement of the AI Act. Therefore, we urge the Commission to take immediate action and:

• Guarantee adequate funding for the AI Office until the next MFF comes into effect. This interim measure should ensure the Office can begin its critical work without resource constraints.
• Negotiate a dedicated budget line within the MFF 2028-2034. This aligns with the strategic importance of the AI Office and prevents reliance on reallocations potentially compromising other programs.

Investing in the AI Office is not just a budgetary decision; it’s an investment in a robust regulatory framework for responsible AI development. By ensuring adequate funding, the Commission can empower the AI Office to effectively oversee the AI Act, safeguard public trust, and enable Europe to remain at the forefront of responsible AI governance.

Domain Definition

A Chemical Weapon is a chemical used intentionally to kill or harm with its toxic properties. Munitions, devices and other equipment specifically designed to weaponize toxic chemicals also fall under the definition of chemical weapons. Chemical agents such as blister agents, choking agents, nerve agents and blood agents have the potential to cause immense pain and suffering, permanent damage and death.¹ After these weapons caused millions of casualties in both world wars, 200 countries signed the Chemical Weapons Convention – enforced by the Organization for the Prohibition of Chemical Weapons (OPCW) – and sought to destroy their chemical stockpiles. With the destruction of the last chemical weapon by the United States in July 2023, the OPCW has declared the end of all official chemical stockpiles.² While small-scale attacks by non-state actors and rogue state actors have occurred over the last fifty years, these are isolated cases. The vast majority of chemical weapons have been eradicated.

Biosecurity encompasses actions to counter biological threats, reduce biological risks, and prepare for, respond to and recover from biological incidents – whether naturally occurring, accidental, or deliberate in origin and whether impacting human, animal, plant, or environmental health. The National Biodefense Strategy and Implementation Plan published by the White House in October 2022 finds biosecurity to be critical to American national security interests, economic innovation, and scientific empowerment.³ In addition, American leadership from both sides of the political spectrum has undertaken significant investments in strengthening biosecurity over the last two decades. Finally, the COVID-19 pandemic has crystallized the threat to American life, liberty, and prosperity from pandemics in the future.

Problem Definition

Artificial intelligence (AI) could reverse the progress made in the last fifty years to abolish chemical weapons and develop strong norms against their use. As part of an initiative at the Swiss Federal Institute for Nuclear, Biological, and Chemical (NBC) Protection, a computational toxicology company was asked to investigate the potential dual-use risks of AI systems involved in drug discovery. The initiative demonstrated that these systems could generate thousands of novel chemical weapons. Most of these new compounds, as well as their key precursors, were not on any government watch-lists due to their novelty.⁴ This is even more concerning in light of the advent of large language model (LLM) based artificial agents. This is because the ability of artificial agents to sense their environment, make decisions, and take actions compounds the unpredictability and risks associated with this kind of research.

On the biological weapons front, cutting-edge biosecurity research, such as gain-of-function research, qualifies as dual-use research of concern – i.e. while such research offers significant potential benefits it also creates significant hazards. For instance, such research may be employed to develop vital medical countermeasures or to synthesize and release a dangerous pathogen. Over the last two decades, the cost of advanced biotechnology has rapidly decreased and access has rapidly expanded through advancements in cheaper and more accessible DNA sequencing, faster DNA synthesis, the discovery of efficient and accurate gene-editing tools such as CRISPR/Cas9, and developments in synthetic biology.⁵

Accompanying these rapid developments are even faster advancements in AI tools used in tandem with biotechnology. For instance, advanced AI systems have enabled several novel practices such as AI-assisted identification of virulence factors and in silico design of novel pathogens.⁶ More general-purpose AI systems, such as large language models, have also enabled a much larger set of individuals to access potentially hazardous information with regard to procuring and weaponizing dangerous pathogens, lowering the barrier of biological competency necessary to carry out these malicious acts.

The threats posed by biological and chemical weapons in convergence with AI are of paramount importance. Sections 4.1 and 4.4 of the White House Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence explicitly outline the potential Chemical, Biological, Radiological and Nuclear (CBRN) threats posed by advanced AI systems.⁷  They can be broadly divided into two categories:

#1. Exponentially Enhanced Capacity to Engineer Deadly Toxins and Biological Weapons

As discussed in the example of the toxicology company above, there is growing concern regarding the potential misuse of molecular machine learning models for harmful purposes. The dual-use application of models for predicting cytotoxicity to create new poisons or employing AlphaFold2 to develop novel toxins has raised alarm. Recent developments in AI have allowed for an expansion of open-source biological design tools (BDTs), increasing access by bad actors.⁸ This creates three kinds of risks:

1. Increased Access to Rapid Identification of Toxins: The MegaSyn AI software used by the toxicology company discussed was able to find 40,000 toxins with minimal digital architecture (namely some programming), open-source data, a 2015 Mac computer and less than six hours of machine time.⁹ This suggests that AI systems may democratize the ability to create chemical weapons, increasing access by non-state actors, rogue states or individuals acting on their own who would otherwise be precluded by insufficient resources. Combined with the use of LLMs and other general-purpose AI tools, the bar for expert knowledge needed to develop chemical weapons has been substantially lowered, further diffusing the ability to identify and release deadly toxins.
2. Discovery of Novel Toxins: An important aspect of the findings from the experiment discussed above is that the AI system not only found VX and other known chemical weapons; it also discovered thousands of completely new putatively toxic substances. This creates serious hazards for chemical defense, as malevolent actors may try to make AI systems develop novel toxins that are less well understood, and for which defensive, neutralizing, or treatment procedures have not yet been developed.
3. AI-Accelerated Development of Biological Design Tools. These tools span different fields such as bio-informatics, genomics, synthetic biology, and others. In essence, these tools allow smaller groups of individuals, with fewer resources, to discover, synthesize, and deploy enhanced pathogens of pandemic potential (PPPs). Critically, these AI systems can amplify risks from gain-of-function research, enabling malevolent actors to make pathogens more deadly, transmissible, and resilient against medical counter-measures.¹⁰ AI assistance can also help bad actors direct such bio-weapons at targets of particular genotypes, races, ethnicities, tribes, families, or individuals, facilitating the conduct of genocide at a potentially global scale.¹¹

#2. Increased Access to Dangerous Information and Manipulation Techniques Through LLMs

Outside the use of narrow AI systems to discover deadly toxic substances, developments in general purpose systems such as large language models may allow malevolent actors to execute many of the other steps needed to deploy a chemical weapon. Essential steps include baseline knowledge of chemistry and biology, access to critical materials and lab infrastructure, and access to means of deploying the weapon (e.g. munitions). LLMs equip malevolent actors with the ability to send deceptive emails and payments to custom manufacturers of chemical and biological materials, access substances through illicit markets, and hire temporary workers to accomplish specialized, compartmentalized tasks around the world. Taken together, these capacities enable the production and deployment of chemical weapons. More narrow AI systems have displayed effectiveness in writing code to exploit technical loopholes in the cybersecurity architecture of several organizations, such as identifying and exploiting zero-day vulnerabilities.¹² Such techniques could be used to target critical bio-infrastructure such as Biosafety Level 3 and 4 Labs (BSL-3 and BSL-4), research laboratories, hospital networks, and more. These practices could enable access to dangerous information or be used to cripple recovery and response to a high-consequence biological incident.

An experiment conducted at MIT demonstrated that students without a technical background were able within 60 minutes to use LLMs to identify four potential pandemic pathogens, explain how they can be generated from synthetic DNA using reverse genetics, supply the names of DNA synthesis companies unlikely to screen orders, identify detailed protocols and how to troubleshoot them, and recommend that anyone lacking the skills to perform reverse genetics engage a core facility or contract research organization.¹³ Other experiments conducted across different settings and time horizons have also demonstrated how large language models can be exploited to access and/or use hazardous information.¹⁴ Traditionally, access to this kind of expertise and information was mediated through established systems (completing a Ph.D. in an advanced field, being hired by a top research laboratory, meeting specified safety and security criteria for conducting sensitive research, etc.). Nowits democratization allows many more individuals, with less skill and less intelligence, to access this knowledge and potentially use it to cause considerable harm.

AI-powered cyberattacks also present a threat to biosecurity and chemical security. Advancements in AI have allowed a wider net of actors to construct more easily cyber exploits that could be used to target cyber-vulnerabilities in water treatment facilities, research labs and containment facilities, to cause widespread harmful chemical or biological exposure. In addition, AI systems may be used to improve the cyber-manipulation techniques used by malevolent actors. Cyber-manipulation encompasses a wide array of practices such as spearphishing, pharming, smishing, vishing, and others intended to deceive, blackmail, mislead, or otherwise compel the victim of such a practice to reveal high-value information. Large language models have demonstrated a considerable capacity to amplify the power of these illegal practices, which could allow malevolent actors to access dangerous biological information or infrastructure by manipulating owners of DNA synthesis companies, prominent academics in the field, and biosecurity professionals.¹⁵ While many large language models have some preliminary guardrails built in to guard against this misuse, several experiments have demonstrated that even trivial efforts can overcome these safeguards.¹⁶ For instance, relabeling of these toxic substances within the data of the model can overcome safeguards which were set up to preclude them from providing dangerous information. Prompt engineering by compartmentalizing (breaking up one dangerous process into several steps which seem innocuous by themselves), as well as faking authority (pretending to be in charge of a government chemical facility), have also yielded success in manipulating these models.¹⁷ 

Policy Recommendations

In light of the significant challenges analyzed in the previous section, considerable attention from policymakers is necessary to ensure the safety and security of the American people. The following policy recommendations represent critical, targeted first steps to mitigating the risks posed by AI in the domains of chemical and biosecurity: :

1. Explicit Requirements to Evaluate Advanced General Purpose AI Systems for Chemical Weapons Use: There is considerable ongoing policy discussion to develop a framework for evaluating advanced general purpose AI systems before and after they are developed and/or deployed, through red-teaming, internal evaluations, external audits and other mechanisms. In order to guard against emerging risks from biological and chemical weapons, it is vital that these evaluations explicitly incorporate a regimen for evaluating a system’s capacity to facilitate access to sensitive information and procedures necessary to develop chemical weapons. This could include the capability of these systems to provide dangerous information as discussed, as well as the capability to deceive, manipulate, access illicit spaces, and/or order illegal financial transactions. In order to prevent malevolent actors from accessing hazardous information and expertise, or further exploiting AI systems to access high-risk infrastructure, it is also critical to set up minimum auditing requirements for these general-purpose systems before launch. These practices could help test and strengthen the safeguards underpinning these systems. Such a requirement could also be incorporated into the existing risk management frameworks, such as the NIST AI Risk Management Framework.
2. Restrict the Release of Model Weights for Systems that Could be Used, or Modified to be Used, to Discover Dangerous Toxins: In order to reduce the ability of malevolent actors to use AI capabilities in production of dangerous chemical toxins, it is critical that both narrow and general-purpose AI systems that are shown to be dangerous in this regard (as well as future iterations of those and similar systems) include significant restrictions on access both for use and to the underlying model weights . Critically, the release of model weights is an irreversible act that eliminates the capacity to restrict use in perpetuity. Accordingly, red-teaming procedures such as those mentioned in the previous recommendation must include extensive assessment to confirm the lack of potential for these dangerous capabilities, and for modification or fine-tuning to introduce these dangerous capabilities, if the developer intends to release the model weights..¹⁸ 
3. Ring-fence Dangerous Information from Being Used to Train Large Language Models. In order to ensure that general-purpose AI systems do not reveal hazardous information, it is vital to require that companies not use this kind of information during training runs to train their AI models. Proactively keeping information that would very likely pose a significant health and/or safety issue to the general public classified using new classification levels and initiatives would significantly reduce these risks.¹⁹  
4. Incorporating AI Threats into Dual Use Research of Concern Guidance and Risk Frameworks: Over the last two decades, considerable policy attention has been devoted to establishing policy frameworks, including guidance and requirements, for biosecurity. However, these frameworks do not currently include policy prescriptions and guidance for unique challenges posed by AI. National-level policy frameworks such as those published by the National Science Advisory Board for Biosecurity (NSABB), the CDC, HHS, DHS, and others must explicitly integrate concerns at the convergence of AI and biosecurity, and establish technical working groups within these bodies populated by experts in both fields to study these risks. Finally, these convergence risks should also be integrated into AI risk frameworks such as the NIST AI RMF. With the exception of the NIST AI RMF, all of these regulatory directives and review regimes were instituted before the exponential development of AI systems seen over the last few years. It is important to update this guidance and include explicit provisions for the use of AI in dual-use biological and chemical research.
5. Expand Know Your Customer (KYC) and Know Your Order (KYO) Requirements. Companies that provide sequencing and synthesis services, research laboratories, and other relevant stakeholders should be required to follow KYC and KYO standards, ensuring that potentially dangerous sequences are kept out of the hands of malevolent actors.²⁰ Regulation should further require standardized, scalably secure synthesis screening methods (such as SecureDNA). These requirements must also include assurance that correspondence pertaining to these services is between human agents and not involving AI systems.
6. Strengthen Existing Capabilities and Capacities for Biodefense. As developments in AI and biotechnology accelerate, it is also vital to ensure that there is considerable capacity to prevent, detect, and respond to high-consequence biological incidents of all kinds. This includes significant investments in early warning and detection, response capacities, interoperability and coordination, national stockpiles of PPEs and other relevant infrastructure, supply-chain resilience, development of medical countermeasures, and accountability and enforcement mechanisms to disincentivize both accidents and intentional misuse.²¹

More general oversight and governance infrastructure for advanced AI systems is also essential to protect against biological and chemical risks from AI, among many other risks. We further recommend these broader regulatory approaches to track, evaluate, and incentivize the responsible design of advanced AI systems:

1. Require Advanced AI Developers to Register Large Training Runs and to “Know Their Customers”: The Federal Government lacks a mechanism for tracking the development and proliferation of advanced AI systems that could exacerbate bio-risk.  To mitigate these risks adequately, it is essential to know what systems are being developed and who has access to them. Requiring registration for the acquisition of large amounts of computational resources for training advanced AI systems, and for carrying out the training runs themselves, would help with evaluating possible risks and taking appropriate precautions. “Know Your Customer” requirements similar to those imposed in the financial services industry would reduce the risk of systems that can facilitate biological and chemical attacks falling into the hands of malicious actors.

2. Clarify Liability for Developers of AI Systems Used in Bio- and Chemical Attacks: It is not clear under existing law whether the developers of AI systems used by others, for example to synthesize and launch a pathogen, would be held liable for resulting harms. Absolving developers of liability in these circumstances creates little incentive for profit-driven developers to expend financial resources on precautionary design principles and robust assessment. Because these systems are opaque and can possess unanticipated, emergent capabilities, there is inherent risk in developing advanced AI systems and systems expected to be used in critical contexts. Implementing strict liability when these systems facilitate or cause harm would better incentivize developers to take appropriate precautions against vulnerabilities, and the risk of use in biological and chemical attacks.

---

^{↩ 1} What is a Chemical Weapon? Organization for the Prohibition of Chemical Weapons.

^{↩ 2} US Completes Chemical Weapons Stockpile Destruction Operations. Department of Defense.

^{↩ 3} National Biodefense Strategy And Implementation Plan. The White House. 

^{↩ 4} Dual Use of Artificial Intelligence-powered Drug Discovery. National Center for Biotechnology Information. National Institutes of Health.

^{↩ 5} The Blessing and Curse of Biotechnology: A Primer on Biosafety and Biosecurity. Carnegie Endowment for International Peace.

^{↩ 6} Assessing the Risks Posed by the Convergence of Artificial Intelligence and Biotechnology.  National Center for Biotechnology Information. National Institutes of Health.

^{↩ 7} Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The White House.

^{↩ 8} Bio X AI: Policy Recommendations For A New Frontier. Federation of American Scientists.

^{↩ 9} AI suggested 40,000 new possible chemical weapons in just six hours. The Verge.

^{↩ 10} The Convergence of Artificial Intelligence and the Life Sciences. Nuclear Threat Initiative.

^{↩ 11} The Coming Threat of a Genetically Engineered ‘Ethnic Bioweapon’. National Review.

^{↩ 12} US adversaries employ generative AI in attempted cyberattack. Security Magazine.

^{↩ 13} Can large language models democratize access to dual-use biotechnology? Computer and Society. https://arxiv.org/abs/2306.03809

^{↩ 14} The Operational Risks of AI in Large-Scale Biological Attacks. RAND Corporation.

^{↩ 15} AI tools such as ChatGPT are generating a mammoth increase in malicious phishing emails. CNBC.

^{↩ 16} NIST Identifies Types of Cyberattacks That Manipulate Behavior of AI Systems. National Institutes of Standards and Technology.

^{↩ 17} Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. Computer Engineering. https://arxiv.org/abs/2305.13860

^{↩ 18} BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13. Computation and Language. https://arxiv.org/abs/2311.00117

^{↩ 19} Artificial Intelligence in the Biological Sciences: Uses, Safety, Security, and Oversight. Congressional Research Service.

^{↩ 20} Preventing the Misuse of DNA Synthesis Technology. Nuclear Threat Initiative.

^{↩ 21} Biosecurity In The Age Of AI. Helena Biosecurity. https://www.helenabiosecurity.org

Today, it is easier than ever to create exploitative deepfakes depicting women in a sexual manner without their consent – and the recently negotiated EU directive combating violence against women could finally bring justice for victims by holding the AI model developers criminally accountable.

Deepfakes refer to AI-generated voices, images, or videos produced without consent, and the most popular type of deepfake, comprising at least 96% of instances, is pornographic. Women and girls make up 99% of victims. Many of these victims will remain unaware that they have been the subject of a deepfake for months after the fact, during which the content garners thousands, sometimes millions, of views.

Given the widespread popularity of deepfake-generating AI systems, the most effective approach to counter deepfakes is for governments to institute comprehensive bans at every stage of production and distribution. Mere criminalization of deepfake production and sharing is insufficient; accountability must extend to the developers, model providers, service providers, and compute providers involved in the process.

Nevertheless, it is not necessarily illegal to create a sexually explicit deepfake in Europe. The final text of the EU AI Act would only require transparency obligations for providers and users of certain AI systems and general-purpose AI models under Article 52. These types of disclosure obligations do very little to mitigate the harms of pornographic deepfakes, given that in the majority of cases the content is consumed with full understanding that it is not truthful. As such, the defamation laws of most EU Member States tend to be equally unhelpful for victims.

The forthcoming directive on combating violence against women could change that. On February 6, 2024, legislators reached a political agreement on rules aimed at combating gender-based violence and protecting its victims. The Directive specifically addresses deepfakes, describing them as the non-consensual production, manipulation, or alteration of material which makes it appear as though another person is engaged in sexual activities. The content must “appreciably” resemble an existing person and “falsely appear to others to be authentic or truthful” (Recital 19).

Publishing deepfakes would be considered a criminal offence under Article 7, as that would constitute using information and communication technologies to make sexually explicit content accessible to the public without the consent of those involved. This offence applies only if the conduct is likely to cause serious harm.

At the same time, aiding, abetting, or inciting the commission of Article 7 would also be a criminal offence under Article 11. As such, providers of AI systems which generate sexual deepfakes may be captured by the directive, since they would be directly enabling the commission of an Article 7 offence. Given that many sites openly advertise their model’s deepfake capabilities and that the training data is usually replete with sexually explicit content, it is difficult to argue that developers and providrs play an insignificant or auxiliary role in the commission of the crime.

The interpretation of Article 11 could be a crucial first step for dismantling the pipeline which fuels sexual exploitation through deepfakes. The broadest reading of Article 11 would imply that developers are subject to corporate criminal liability.

One important hurdle is that corporate criminal liability does not apply uniformly across Europe, with some Member States recognizing corporations as entities capable of committing crimes, while others do not. Nevertheless, the application of Article 11 in at least some jurisdictions would be a tremendous step towards stopping the mass production of sexual deepfakes. Afterall, jurisdiction is established based on territory, nationality, and residence according to Article 14.

The directive also briefly addresses the role of hosting and intermediary platforms. Recital 40 empowers Member States to order hosting service providers to remove or disable access to material violating Article 7, encouraging cooperation and self-regulation through a code of conduct. While this may be an acceptable level of responsibility for intermediaries, self-regulation is entirely inappropriate for providers who constitute the active and deliberate source of downstream harm.

The final plenary vote is scheduled for April. The capacity for this directive to protect women and girls from being exploited through harmful deepfakes rides on whether the companies commercializing this exploitation are also held criminally liable.

Organization: Future of Life Institute

Point of Contact: Hamza Tariq Chaudhry, US Policy Specialist. hamza@futureoflife.blackfin.biz 

We would like to thank the Office of Management and Budget (OMB) for the opportunity to provide comments on OMB–2023–0020, or the Memorandum on ‘Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence’ (hereafter referred to as ‘the Memorandum’). The Future of Life Institute (FLI) has a long-standing tradition of work on AI governance to mitigate the risks and maximize the benefits of artificial intelligence. For the remainder of this Request for Comment (RfC) document, we provide a brief summary of our organization’s work in this space, followed by substantive comments on the Memorandum. The ‘substantive comments’ section provides responses to the questions outlined in the RfC. The ‘miscellaneous comments’ section offers general comments outside the scope of the questions outlined in the Federal Register. We look forward to continuing this correspondence and being a resource for the OMB’s efforts in this space in the months and years to come.

About the Organization

The Future of Life Institute (FLI) is an independent nonprofit organization with the goal of reducing large-scale risks and steering transformative technologies to benefit humanity, with a particular focus on artificial intelligence. Since its founding ten years ago, FLI has taken a leading role in advancing key disciplines such as AI governance, AI safety, and trustworthy and responsible AI, and is widely considered to be among the first civil society actors focused on these issues. FLI was responsible for convening the first major conference on AI safety in Puerto Rico in 2015, and for publishing the Asilomar AI principles, one of the earliest and most influential frameworks for the governance of artificial intelligence. FLI is the UN Secretary General’s designated civil society organization for recommendations on the governance of AI and has played a central role in deliberations regarding the EU AI Act’s treatment of risks from AI. FLI has also worked actively within the United States on legislation and executive directives concerning AI. Members of our team have contributed extensive feedback to the development of the NIST AI Risk Management Framework, testified at the Senate Insight Forums, participated in the UK AI Summit, and helped connect leading experts in the policy and technical domains to policymakers across the US government.

FLI’s wide-ranging work on artificial intelligence can be found at www.futureoflife.blackfin.biz.

---

Substantive Comments

Definitions and best practices

Comments in Response to Questions 5 and 6 from the Federal Register: Definitions of and best practices regarding safety-impacting and rights-impacting AI.¹

The Memorandum establishes a clear minimum threshold for safety (Section 5,c,iv) that must be attained before agencies are allowed to use AI systems, applicable to both systems being used presently and those intended for use in the future.² The requirements for these agencies – which include impact assessments, real-world testing of AI systems before deployment, independent evaluations and periodic post-deployment testing – are a positive step towards minimizing the safety risks from government use of AI models.

We would, however, welcome further details for agencies on the periodic reviews that occur post-deployment to specify that these reviews would also include red-teaming and other auditing processes that make up portions of the pre-deployment review process. In addition, while we appreciate the inclusion of  language prohibiting agencies from using AI systems in cases where ‘the benefits do not meaningfully outweigh the risks’, we invite the OMB to support this language with quantitative examples, as risks may capture both probability and magnitude of harm, especially in the case of safety concerns. For instance, even if the probability of any given risk is found to be considerably lower than that of potential benefit, the magnitude of a risk (e.g., a bio-weapon attack) may be so high that it overrides the benefit despite being of low probability. Agencies should be required to establish, subject to public comment and external review, risk tolerances for activities for which use of AI systems is anticipated, including unacceptable risks to individuals, communities, and society that would disqualify the system from adoption. Establishing these thresholds prior to testing and adoption would help prevent drift in risk tolerance that could gradually rise to insufficient levels.

The Memorandum provides adequate definitions for two categories of potential harm posed by AI systems – safety-impacting AI systems and rights-impacting AI systems. FLI, which predominately focuses on AI safety, supports the broader definition of safety-impacting AI systems offered in the Memorandum, which captures a more expansive set of AI models and does not rely on technical thresholds. We believe this best positions the executing Agencies to exercise appropriate oversight over use of AI models. In addition, we are pleased to see that under the proposed definition, many models are presumed to be safety-impacting (Section 5,b). This is critical as it relieves relevant agencies of administrative burdens and time delays that would otherwise occur in evaluating each system with risk assessments, instead allowing them to devote more time and resources to setting up adequate guardrails. On the same token, we are pleased that additional risk assessments can be conducted to expand the scope of systems receiving due scrutiny. 

Finally, when it comes to ‘use of AI’, we support efforts to include cases in the Memorandum of procurement in addition to direct use (Section 5, d). However, the language of the Memorandum currently forwards guidance on procurement and contracts not as a set of requirements but as a set of recommendations. It is imperative that the OMB set up robust requirements for government purchasing of AI systems that mirror requirements on direct use, ensuring that procurement of AI systems includes consistent, robust evaluation to protect the safety and rights of the American public. This has the potential to minimize harm from government use of AI, and to inform best practices for the private sector, where most of that state-of-the-art models are created.

Chief AI Officer and AI Governance Body

Comments in Response to Questions 1 and 2 from the Federal Register: Role of Chief AI Officer and the benefits and drawbacks of central AI governance body.³

We agree that effective oversight of AI adoption by government agencies should rely on AI governance bodies within each agency to coordinate and supervise AI procurement and use across the broad functions of the agency.This structure facilitates oversight and accountability  to ensure that minimum requirements as set out in the Memorandum are met by each agency writ large, while giving different offices within each agency the capability to exercise their mandate when it comes to specific use cases. In addition, we believe such a body can facilitate effective communication across different offices, bureaus and centers within the agency to ensure that poor communication feedback does not lead to under-reporting of use cases or use of AI that could lead to potential harm. Finally, we believe such a body would appropriately empower the Chief AI Officer (CAIO) to exercise their mandate as specified in the Memorandum.

However, we contend that this “hub and spoke” structure of a centralized AI governance body coordinating and overseeing domain-specific AI governance should be implemented on a whole-of-government level. In other words, we believe that just as there are benefits to having a new central body within each agency that helps enforce requirements laid out within the Memorandum, these bodies themselves would benefit from a single governance body that has representation and oversight across different agencies. This would facilitate interagency coordination, provide a central hub of expertise to advise agencies where appropriate, avoid costly redundancies in efforts by various agencies, and provide a body to inform and evaluate government AI adoption where domain-specific agency jurisdiction is not clear.

Information for Public Reporting

Comments in Response to Question 8 from the Federal Register: Nature of information that should be publicly reported by agencies in use case inventories.⁴

While we welcome provisions within the Memorandum which require annual reporting of use cases of covered AI systems by the relevant agencies (Section 3, a), we are concerned that further elaboration is not provided by the OMB on the details of these use case inventories. We believe that the public should have access to information on the full results of the impact assessments, real-world testing, independent evaluations, and periodic human reviews, wherever possible. Where it is not possible to provide this information in full, we believe it is vital to provide redacted iterations of these documents upon the filing of a Freedom of Information Act (FOIA) request. Secondly, considering that there is some precedent of agencies neglecting to report all use cases in the past, we believe that the Memorandum would benefit from having explicit provisions to guard against under-reporting of use cases. This could, for instance, include guidance for Inspectors General to audit these use cases periodically within their respective agencies. Finally, while we recognize this as a positive first step towards creating transparency in use cases, we emphasize that this does not ensure sufficient accountability in and of itself, and will require further guidance and requirements on empowering the OMB and the CAIOs, and other relevant authorities, to take against violations of use case guidance set up in the Memorandum.

Miscellaneous Comments

Comments on Scope

Section 2 (‘Scope’) explicitly exempts the intelligence community (‘covered agencies’, Section 2, a) and cases where AI when it is used as a component of a national security system (‘applicability to national security systems’, Section, c). As the Memorandum is intended to minimize the risks of government use of AI systems, we believe it is critical to establish robust requirements for the intelligence and defense communities, as these are likely to be the highest risk cases of government AI use with the greatest potential harm, and hence the most urgent need for scrutiny. Where it is within the remit of the OMB to set up requirements within these domains, we ask that they urgently do so.

Comments on Definitions

We are pleased to see that Section 6 (‘Definitions’) outlines an expansive definition of “artificial intelligence” that is broader than the definition offered in the AI Executive Order. In addition, we support that the Memorandum’s description of AI systems encompasses all those across different ranges of autonomous behavior, technical parameters and human oversight. However, we believe that it is vital to ensure that the definition of AI employed in this section is treated as an ‘or’ definition as opposed to an ‘and’ definition. In other words, we believe that any system which fulfills any of these criteria should fall within the definitional scope of AI. For the same reason, we are concerned that the definition of ‘dual-use foundation models’ mirrors the definition included in the AI Executive Order, which offers an ‘and’ definition leading to very few models coming under definitional scope, and potentially excluding those which pose safety risks but do not meet other criteria.⁵

The Memorandum also employs the AI Executive Order definition for ‘red-teaming’.⁶ While this definition outlines what red-teaming would cover, it does not provide any detail on how rigorous this red-teaming must be, and for what period within the lifecycle of the AI system. We support further clarification from the OMB in this regard to ensure that red-teaming as defined in guidance adequately tests models for safety harms for the duration of their procurement and use.

We endorse the OMB’s decision to establish a broad definition for what would count as ‘risks from the use of AI’ as well as the expansive definition of ‘safety-impacting AI’. However, we recommend the addition of loss of control from use of AI systems to the considerable list of risk factors identified in the definition of ‘safety-impacting AI’.

Comments on Distinguishing between Generative and Other AI

We believe that all advanced AI systems, whether they are generative or otherwise, should be subject to appropriate requirements to ensure safety. Hence, we are pleased to see that, in a slight divergence from the AI Executive Order, the Memorandum bases requirements on potential harms from AI and does not distinguish between generative AI and other AI systems.

---

^{↩ 1} 5. Are there use cases for presumed safety-impacting and rights-impacting AI (Section 5 (b)) that should be included, removed, or revised? If so, why?

6. Do the minimum practices identified for safety-impacting and rights-impacting AI set an appropriate baseline that is applicable across all agencies and all such uses of AI? How can the minimum practices be improved, recognizing that agencies will need to apply context-specific risk mitigations in addition to what is listed?

^{↩ 2} We are particularly pleased to see that the scope of this Memorandum applies not just to use and application of AI systems in the future, but also those currently in use by relevant agencies.

^{↩ 3} 1. The composition of Federal agencies varies significantly in ways that will shape the way they approach governance. An overarching Federal policy must account for differences in an agency’s size, organization, budget, mission, organic AI talent, and more. Are the roles, responsibilities, seniority, position, and reporting structures outlined for Chief AI Officers sufficiently flexible and achievable for the breadth of covered agencies?

2. What types of coordination mechanisms, either in the public or private sector, would be particularly effective for agencies to model in their establishment of an AI Governance Body? What are the benefits or drawbacks to having agencies establishing a new body to perform AI governance versus updating the scope of an existing group (for example, agency bodies focused on privacy, IT, or data)?

^{↩ 4} 8. What kind of information should be made public about agencies’ use of AI in their annual use case inventory?

^{↩ 5} Section 3 of the AI Executive Order defines such as model in the following way: “dual-use foundation model” means an AI model that is trained on broad data; generally uses self-supervision; contains at least tens of billions of parameters; is applicable across a wide range of contexts; and that exhibits, or could be easily modified to exhibit, high levels of performance at tasks that pose a serious risk to security, national economic security, national public health or safety, or any combination of those matters.” (Emphasis added).

^{↩ 6} Section 3 of the AI Executive Order defines red-teaming as: The term “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI. Artificial Intelligence red-teaming is most often performed by dedicated “red teams” that adopt adversarial methods to identify flaws and vulnerabilities, such as harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with the misuse of the system.

Response to Request for Information (RFI NIST-2023-0009-0001) Related to NIST’s Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence (Sections 4.1, 4.5, and 11)

Organization: Future of Life Institute

Point of Contact: Hamza Tariq Chaudhry, US Policy Specialist. hamza@futureoflife.blackfin.biz 

About the Organization

The Future of Life Institute (FLI) is an independent nonprofit organization with the goal of reducing large-scale risks and steering transformative technologies to benefit humanity, with a particular focus on artificial intelligence. Since its founding, FLI has taken a leading role in advancing key disciplines such as AI governance, AI safety, and trustworthy and responsible AI, and is widely considered to be among the first civil society actors focused on these issues. FLI was responsible for convening the first major conference on AI safety in Puerto Rico in 2015, and for publishing the Asilomar AI principles, one of the earliest and most influential frameworks for the governance of artificial intelligence, in 2017. FLI is the UN Secretary General’s designated civil society organization for recommendations on the governance of AI and has played a central role in deliberations regarding the EU AI Act’s treatment of risks from AI. FLI has also worked actively within the United States on legislation and executive directives concerning AI. Members of our team have contributed extensive feedback to the development of the NIST AI Risk Management Framework, testified at Senate AI Insight Forums, participated in the UK AI Summit, and connected leading experts in the policy and technical domains to policymakers across the US government.

---

Executive Summary

We would like to thank the National Institute of Standards and Technology (NIST) for the opportunity to provide comments regarding NIST’s assignments under Sections 4.1, 4.5 and 11 of the Executive Order Concerning Artificial Intelligence (Sections 4.1, 4.5, and 11). The Future of Life Institute (FLI) has a long-standing tradition of work on AI governance to mitigate the risks and maximize the benefits of artificial intelligence. In NIST’s implementation of the Executive Order 13960 on “Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government” (EO), we recommend consideration of the following:

• Military and national security AI use-cases should not be exempt from guidance. Military and national security AI use-cases, despite exemptions in the Executive Order, should not be beyond NIST’s guidance, considering their potential for serious harm. Given their previous work, NIST is well-positioned to incorporate standards for military and national security into their guidelines and standards.
• A companion resource to the AI Risk Management Framework (RMF) should explicitly characterize minimum criteria for unacceptable risks. NIST should guide the establishment of tolerable risk thresholds. These thresholds should include guidelines on determining unacceptable risk and outline enforcement mechanisms and incentives to encourage compliance.
• Responsibility for managing risks inherent to AI systems should fall primarily on developers. The NIST companion resource for generative AI should define roles for developers, deployers, and end-users in the assessment process. Developers, end-users, and deployers should all work to mitigate risk, but the responsibility of developers is paramount, considering their central role in ensuring the safety of systems.
• Dual-use foundation models developed by AI companies should require external red-teaming. External red-teams are essential for encouraging comprehensive and unbiased assessments of AI models. NIST should establish standards to ensure that external auditors remain independent and aligned with best practices.
• The NIST companion resource for generative AI should include specific guidance for AI models with widely available model weights. Safeguards designed to mitigate risks from dual-use foundations models with widely available weights can be easily removed and require specific standards to ensure security.
• Embedded provenance on synthetic content should include developer and model information. Including information on synthetic content about the developer and system of origin would better inform consumers and incentivize developers to prioritize safety from the design phase.
• NIST should adopt a less restrictive definition of “dual-use foundation models.” Switching from a restrictive definition (using ‘and’) to a more expansive definition (using ‘or’) as stated in the EO would enable NIST to bring all models of concern within its purview.

We look forward to continuing this correspondence and to serve as a resource for NIST efforts pertaining to AI in the months and years to come.

---

Recommendations

1. Military and national security use-cases  

Standards for national security and military are not beyond the remit of NIST. AI systems intended for use in national security and military applications present some of the greatest potential for catastrophic risk due to their intended use in critical, often life-or-death circumstances. While the EO exempts national security and military AI from most of its provisions, NIST has previously established standards¹ related to national security, including standards for chemical, biological, radiological, nuclear, explosive (CBRNE) detection, personal protective equipment (PPE), and physical infrastructure resilience and security. Given this precedent, NIST can and should update the AI RMF and future companion pieces to includestandards applicable to national security and military uses of AI. Specifically, NIST can play a vital role in mitigating risks presented by these systems by, inter alia, working with the Defense Technology Security Administration (DTSA) and the Office of Science and Technology to instate standards for procurement, development and deployment of AI technologies. Considering the sizable impact malfunction, misuse, or malicious use of military or national security AI systems could entail, such standards should be at least as rigorous in assessing and mitigating potential risks as those developed for civilian AI applications.

2. Addressing AI RMF gaps   

NIST should provide guidance on identifying unacceptable risks. The AI risk management framework lacks guidance on tolerable risk thresholds. As a result, developers of potentially dangerous AI systems can remain in compliance with the AI RMF despite failure to meaningfully mitigate substantial risks, so long as they document identification of the risk and determine that risk to be acceptable to them. Accordingly, companies can interpret risk solely in terms of their interests – tolerable risk may be construed as risks that are tolerable for the developer, even if those risks are unacceptable to other affected parties. The ability to make internal determinations of tolerable risk without a framework for evaluating externalities overlooks the potential impact on government, individuals, and society. NIST should revise the AI RMF, introducing criteria for determining tolerable risk thresholds. This revision should incorporate evaluations of risk to individuals, communities, and society at each stage of the assessment process, and these revisions should be applied to all relevant companion resources.

Enforcement mechanisms and structural incentives are necessary. While industries may voluntarily adopt NIST standards, we cannot rely on AI companies to continue to self-regulate. The significance of these standards warrants explicit commitment through structured incentives and enforcement measures. To encourage the adoption of these standards, NIST should offer independent evaluation of systems and practices for compliance with their framework, provide feedback, and provide compliant parties with a certificate of accreditation that can demonstrate good faith and strengthen credibility with the public and other stakeholders.

Guidelines must set clear red-lines to halt or remediate projects. NIST should internally define minimum red-lines and encourage AI companies to predetermine additional red-lines for each assessment. Failure to stay within these limits should prevent the project from progressing or mandate remediation. Red-lines should encompass material risks of catastrophic harm and significant risks related to the ease and scale of misinformation, disinformation, fraud, and objectionable content like child sexual abuse material and defamatory media. Such predetermined, explicit thresholds for halting a project or taking remediation efforts will prevent movement of safety and ethical goalposts in the face of potential profits by companies, increasing the practical impact of the AI RMF’s extensive guidance on assessment of risk.

3. AI developer responsibility  

The NIST companion resource for generative AI should define clear roles for developers, deployers, and end-users in the assessment process. All of these parties should take steps to mitigate risks to the extent possible, but the role of the developer in proactively identifying, addressing, and continuously monitoring potential risks throughout the lifecycle of the AI system is paramount. This should include (but is not limited to) implementing robust risk mitigation strategies, regularly updating the system to address new vulnerabilities, and transparently communicating with deployers and end-users about the limitation and safe usage guidelines of the system.

Compared to downstream entities, developers have the most comprehensive understanding of how a system was trained, its behavior, implemented safeguards, architectural details, and potential vulnerabilities. This information is often withheld from the public for security or intellectual property reasons, significantly limiting the ability of deployers and end-users to understand the risks these systems may present. For this reason, deployers and end-users cannot be reasonably expected to anticipate, mitigate, or compensate harms to the extent that developers can.

Deployers implementing safety and security by design, and thus mitigating risks at the outset prior to distribution, is more cost-effective, as the responsibility for the most intensive assessment and risk mitigation falls primarily on the handful of major companies developing advanced systems, rather than imposing these requirements on the more numerous, often resource-limited deployers. This upstream-approach to risk-mitigation also simplifies oversight, as monitoring a smaller group of developers is more manageable than overseeing the larger population of deployers and end-users. Furthermore, the ability of generative AI to trivialize and scale the proliferation of content makes dealing with the issue primarily at the level of the end user infeasible and may also necessitate more privacy-invasive surveillance to implement effectively.

Developer responsibility does not fully exempt deployers or end-users from liability in cases of intentional misuse or harmful modifications of the system. A framework including strict, joint and several liability, which holds all parties in the value chain accountable within their respective liability scopes, is appropriate. Failure by a developer to design a system with sufficient safeguards that cannot be easily circumvented should be considered akin to producing and distributing an inherently unsafe or defective product.

4. External red-teaming of dual-use foundation models  

External red-teaming should be considered a best practice for AI safety. While many AI developers currently hire external teams with specialized knowledge to test their products, relying solely on developers to select these teams is insufficient due to inadequate standardization, conflicts of interest, and lack of expertise.

Ideally, the government would establish the capacity to serve in this role. However, in situations where government-led red-teaming is not feasible, alternative mechanisms must be in place. NIST should move to establish a criteria to assess external auditors for their expertise and independence.² These mechanisms could be implemented as an official certification displayed on the product’s website, signifying that the model has passed testing by an approved entity. This approach not only enhances safety but also fosters transparency and public trust.

Ensuring comprehensive safety assessments requires red-teams to have access to the exact model intended for deployment, along with detailed information on implemented safeguards and internal red-teaming results. External testers are typically given “black-box” access to AI models via API access.³ While fine-tuning can still be supported via API access, this approach at least somewhat limits their testing abilities to prompting the system and observing its outputs. While this is a necessary part of the assessment process, it is not sufficient and has shown to be unreliable in various ways.⁴ Conversely, structured access provides testers with information that allows them to execute stronger, more comprehensive adversarial attacks.⁵ Many companies oppose providing complete access to their models due to concerns about intellectual property and security leaks. To mitigate these concerns, we recommend that NIST establish physical and contractual standards and protocols to enable secure model access such as on-site testing environments and nondisclosure agreements. To ensure that external auditors are conducting tests in accordance with these standards and practices, these should be conducted by the government or other approved entities.

Red-teams should be afforded ample time, resources, and access for comprehensive testing. A multi-stage red-teaming process including data, pre-training, model, system, deployment, and post-deployment phases is needed. Access to training data, for example, could foster transparency and enable pathways for the enforcement of copyright law. Furthermore, developers should be encouraged to proactively engage with deployers to understand the use-cases of their products and inform external auditors so that they may tailor their testing strategies effectively.

Finally, AI companies should be encouraged to establish mechanisms for the continuous identification and reporting of vulnerabilities post-deployment. Many companies have created pipelines for these processes.⁶⁷ NIST should consider providing guidelines to encourage consistency and standardization.

5. Safety limitations of AI models with widely available model weights  

The NIST companion resource on generative AI should include recommendations on evaluating the risks of releasing and developing models with widely available model weights. With current technologies and architectures, removing safeguards from AI models with widely available model weights through fine-tuning is relatively trivial.⁸ This makes it intractable to set or enforce guidelines for developers who build on open-source models. This ease of removal has enabled the proliferation of harmful synthetic materials.⁹ 

6. Inclusion of developer information in synthetic content  

Embedded information on synthetic content should include information about the developer and system of origin. Much attention has been paid in recent months to the potential for synthetic content to contribute to the spread of mis- and disinformation and non-consensual sexual imagery. The proliferation of synthetic content also carries significant national security risks, including the use of synthetic blackmail or spearphishing against high-ranking officials and the creation of fake intelligence, which could introduce serious vulnerabilities. Some generative AI systems may lack sufficient safeguards, making them more prone to these malicious uses, but detecting these vulnerabilities and holding their developers accountable for rectifying them is at present extremely challenging.

Labeling and watermarking techniques have been proposed as one possible method for verifying the authenticity or synthetic nature of content, and Section 4.5(a) of the EO tasks the Department of Commerce with developing or identifying existing tools, standards, methods, practices, and techniques for detecting, labeling, and authenticating synthetic content. We recommend that standards for watermarking or other embedded information should include information detailing the developer and system of origin. Such measures would incentivize developers to prioritize safety from the design phase, facilitate identification of systems especially vulnerable to creation of untoward content, and streamline the identification and tracking of problematic synthetic content back to its creators to impose liability for harms where appropriate. Given the stakes of the issues raised by synthetic content, the emphasis on safety and accountability should take precedent over concerns about the economic feasibility of implementation. That said, any additional economic burden for embedding system and developer of origin information would likely be negligible relative to embedding information relating to the authenticity of the content alone.

7. Definition of “dual-use foundation models”  

The EO defines “dual-use foundation model” to mean “an AI model that is trained on broad data; generally uses self-supervision; contains at least tens of billions of parameters; is applicable across a wide range of contexts; and that exhibits, or could be easily modified to exhibit, high levels of performance at tasks that pose a serious risk to security, national economic security, national public health or safety, or any combination of those matters, such as by:

(i) substantially lowering the barrier of entry for non-experts to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear (CBRN) weapons;

(ii) enabling powerful offensive cyber operations through automated vulnerability discovery and exploitation against a wide range of potential targets of cyber attacks; or

(iii) permitting the evasion of human control or oversight through means of deception or obfuscation.”

It should be noted, however, that the broad general purpose capabilities of “foundation” models inherently render them dual-use technologies. These models can often possess latent or unanticipated capabilities, or be used in unanticipated ways that present substantial risk, even if they do not obviously exhibit performance that poses “a serious risk to security, national economic security, national public health or safety, or any combination of those matters” upon initial observation. Furthermore, models that are not developed in accordance with the described characteristics (i.e. trained on broad data, generally using self-supervision, containing at least tens of billions of parameters, and applicable across a wide range of contexts) that exhibit, or can be easily modified to exhibit, high levels of performance at tasks that pose those serious risks should nonetheless be considered dual-use. Novel architectures for AI systems that can be trained on more specialized datasets or can effectively use fewer parameters, for instance, should fall under the definition if it is evident that they can pose serious risks to national security and public health. Models of this inherently risky architecture AND models that pose an evident risk to security and/or health should be subject to guidance and rigorous safety standards developed by NIST and other agencies pursuant to the EO and beyond.

A slight modification to the EO’s definition of “dual-use foundation models,” as follows, could accommodate this more inclusive concept of dual-use to appropriately scope NIST’s guidance for ensuring the safety of AI systems:

“[…]an AI model that is trained on broad data; generally uses self-supervision; contains at least tens of billions of parameters; is applicable across a wide range of contexts; and or that exhibits, or could be easily modified to exhibit, high levels of performance at tasks that pose a serious risk to security, national economic security, national public health or safety, or any combination of those matters, such…”

8. Global engagement and global military use-cases  

Inclusion of strategic competitors: While we welcome efforts through NIST as outlined in Sec. 11 of the EO to advance global technical standards for AI development, we are concerned about the nature of engagement on this issue restricted to ‘key international allies and partners’. Cognizant of political realities, we ask that NIST also engage with strategic competitors on global technical standards, in particular those states which are considered to be leaders in AI development, such as the PRC. Without engaging with these strategic competitors, any global standards developed will suffer from a lack of enforcement and global legitimacy. Conversely, standards developed in cooperation with strategic competitors would likely strengthen the legitimacy and enforcement potential of technical standards. Moreover, it is in the United States’ national security interests for adversaries’ AI to behave more reliably and predictably, and for these systems to remain under proper human control, rather than malfunctioning to escalate situations without human intent or otherwise cause substantial harm that could diffuse beyond their borders.

The exclusion of military AI use-cases will hinder progress on developing global technical standards generally: As the EO outlines, developing global technical standards on civilian AI development and deployment is vital to reaching a global agreement on use of AI. However, considering the blurry boundary between AI developed and deployed for civilian versus military use, we are concerned that a standards agreement on civilian AI alone will likely be difficult without discussing basic guardrails regarding military development and use of AI. This is because with the most advanced AI systems, distinguishing between military and civilian use cases is becoming and will continue to become increasingly difficult, especially considering their general-purpose nature. Mistrust regarding military AI endeavors is likely to impede the international cooperation necessary to ensure global safety in a world with powerful AI systems, including in civilian domains. Adopting basic domestic safety standards for military use of AI, as recommended in #1 (“Military and national security use-cases”), would reduce the risk of catastrophic failure of military systems and inadvertent escalation between strategic competitors, encourage international adoption of military AI safety and security standards, and foster the trust necessary to encourage broader civilian global AI standards adoption. Hence, we reiterate the request that NIST work actively with the Department of State, the Assistant to the President for National Security and other relevant actors as specified in Section 11, to clarify how its AI safety and security standards can be applied in the military context, especially with respect to models that meet the EO definition of ‘dual-use foundation models’.

Closing Remarks

We appreciate the efforts of NIST to thoughtfully and comprehensively carry out its obligations under the AI EO and are grateful for the opportunity to contribute to this important effort. We hope to continue engaging with this project and subsequent projects seeking to ensure AI does not jeopardize the continued safety, security, and wellbeing of the United States.

---

^{↩ 1} Public Safety – National Security Standards. National Institute of Standards and Technology. Accessed at: https://www.nist.gov/national-security-standards 

^{↩ 2} Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. 2022. Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’22). Association for Computing Machinery, New York, NY, USA, 557–571. https://doi.org/10.1145/3514094.3534181

^{↩ 3} METR. (March 17, 2023). Update on ARC’s recent eval efforts. Model Evaluation and Threat Research.

^{↩ 4} Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., …and Hadfield-Menell, D. (2024). Black-Box Access is Insufficient for Rigorous AI Audits. arXiv preprint arXiv:2401.14446.

^{↩ 5} Bucknall, B. S., and Trager, R. F. (2023). Structured Access For Third-party Research On Frontier ai models: investigating researchers model access requirements. Oxford Martin School AI Governance Initiative.

^{↩ 6} Company Announcement. (July, 2023). Frontier Threats Red Teaming for AI Safety. Anthropic. 

^{↩ 7} Blog. OpenAI Red Teaming Network. OpenAI.

^{↩ 8} Qi, X., Zeng, Y., Xie, T., Chen, P. Y., Jia, R., Mittal, P., & Henderson, P. (2023). Fine-tuning aligned language models compromises safety, even when users do not intend to!. arXiv preprint arXiv:2310.03693. Accessed at: https://arxiv.org/abs/2310.03693

^{↩ 9} Weiss, B. and Sternlicht, A. (January 8, 2024). Meta and OpenAI have spawned a wave of AI sex companions—and some of them are children.

Request for Comments on Implementation of Additional Export Controls: Certain Advanced Computing Items; Supercomputer and Semiconductor End Use (RIN 0694–AI94)

Organization: Future of Life Institute

Point of Contact: Hamza Tariq Chaudhry, US Policy Specialist. hamza@futureoflife.blackfin.biz 

We would like to thank the Bureau of Industry and Security for the opportunity to provide comments on the October 7 Interim Final Rule (IFR), or the Rule on ‘Implementation of Additional Export Controls: Certain Advanced Computing Items; Supercomputer and Semiconductor End Use’ (hereafter referred to as ‘AC/S IFR’). The Future of Life Institute (FLI) has a long-standing tradition of work on AI governance to mitigate the risks and maximize the benefits of artificial intelligence. For the remainder of this Request for Comment (RfC) document, we provide a brief summary of our organization’s work in this space, followed by comments on the AC/S IFR. Our primary comment responds to the RfC on developing technical solutions to exempt items otherwise classified under ECCNs 3A090 and 4A090, and recommends a pilot program for a technical solution. The comment includes arguments for how the pilot program could help improve BIS export controls and mitigate threats to US economic and national-security interests. In the final section, we offer general comments to the AC/S IFR.

We look forward to continuing this correspondence and to serve as a resource for BIS efforts pertaining to AI in the months and years to come.

About the Organization

The Future of Life Institute (FLI) is an independent nonprofit organization with the goal of reducing large-scale risks and steering transformative technologies to benefit humanity, with a particular focus on artificial intelligence. Since its founding ten years ago, FLI has taken a leading role in advancing key disciplines such as AI governance, AI safety, and trustworthy and responsible AI, and is widely considered to be among the first civil society actors focused on these issues. FLI was responsible for convening the first major conference on AI safety in Puerto Rico in 2015, and for publishing the Asilomar AI principles, one of the earliest and most influential frameworks for the governance of artificial intelligence, in 2017. FLI is the UN Secretary General’s designated civil society organization for recommendations on the governance of AI and has played a central role in deliberations regarding the EU AI Act’s treatment of risks from AI. FLI has also worked actively within the United States on legislation and executive directives concerning AI. Members of our team have contributed extensive feedback to the development of the NIST AI Risk Management Framework, testified at Senate AI Insight Forums, participated in the UK AI Summit, and connected leading experts in the policy and technical domains to policymakers across the US government.

FLI’s wide-ranging work on artificial intelligence and beyond can be found at www.futureoflife.blackfin.biz.

---

Primary Comment on Hardware Governance

On the Request for Comment on Developing technical solutions to exempt items otherwise classified under ECCNs 3A090 and 4A090.

We welcome the request for technical solutions on this issue. FLI has recently been involved in multiple initiatives to create and improve technical solutions for the governance of AI hardware, including semiconductors. In this primary comment, we offer arguments in favor of technical solutions for hardware governance, and introduce a new project from FLI which seeks to improve on-chip governance.

Arguments for Technical Solutions for Hardware Governance

Technical solutions for hardware governance, and specifically chip governance, offer many benefits that can supplement top-down export controls as currently instated by BIS.

Generic Export Controls More Vulnerable to Lack of Enforcement than Hardware Governance

Export controls, especially those with a wide and expansive purview, are likely to suffer from serious gaps in enforcement. A growing informal economy around chip smuggling has already emerged over the last few years, and it is likely to grow as BIS rules become more expansive. A solution focused on hardware governance is less liable to this gap in enforcement.

Hardware Governance as Less Blunt Instrument and Less Likely to Hurt US Economic Interests

Export controls most directly target state actors, leading to conflation between ‘actor’ vs ‘application’ that may foreclose benefits and exacerbate risks to United States interests. For instance, broadly-applied export controls targeted at the People’s Republic of China (PRC) do not distinguish between harmless and harmful use cases within the PRC, the former of which can be economically beneficial to the United States and reduce geo-strategic escalation. For instance, relaxing restrictions on chip exports to demonstrably low-risk customers in China helps drive the economic competitiveness of US firms. These economic benefits are integral to guaranteeing continuing US leadership in the technological frontier, and help preserve global stability. Hardware governance, a more targeted instrument, side-steps these issues with export controls, focusing on applications as opposed to actors.

Hardware Governance is Privacy Preserving and Compatible with Existing Chips Technology

New and innovative hardware governance solutions are completely compatible with the current state of the art chips sold by leading manufacturers. All relevant hardware (H100s, A100s, TPUs, etc.) have some form of “trusted platform module”, a hardware device that generates random numbers, holds encryption keys, and interfaces with other hardware modules to provide security.  Some new hardware (H100s in particular) has an additional hardware “secure enclave” capability, which prevents access to chosen sections of memory at the hardware level. TPM and secure enclaves already serve to prevent iPhones from being “jailbroken,” and to secure biometric and other highly sensitive information in modern phones and laptops. Hence, a technical solution to hardware governance would not impose serious costs on leading chip companies to modify the architecture of chips currently in inventory or in production. Critically, as the project described below demonstrates, it is possible to use these technical solutions without creating back-channels that would harm the privacy of end-users of the chip supply chain. Accordingly, hardware governance solutions such as the one proposed below are less likely to face resistance to implementation from concerned parties.

Technical Project – Secure Hardware Solutions for Safe AI Deployment

Background

Modern techniques in cryptography and secure hardware technology provide the building blocks to create verifiable systems that can enforce AI governance policies. For example, an un-falsifiable cryptographic proof can be created to attest that a model comes from the application of a specific code on a specific dataset. This could prevent copyright issues, or prove that a certain number of training epochs were carried out for a given model, verifying whether a threshold in compute has or has not been breached. The field of secure hardware has been evolving and has reached a stage where it can be used in production to make AI safer. While initially developed for users’ devices (e.g. Apple’s use of secure enclaves to securely store and process biometric data on iPhones), large server-side processors have become mature enough to tackle securely governed AI workloads.

While recent cutting-edge AI hardware, such as Intel Xeon with Intel SGX or Nvidia H100s with Confidential Computing, possess the hardware features to implement technical mechanisms for AI governance , few projects have emerged to leverage them to build AI governance tooling. The Future of Life Institute has partnered with Mithril Security, a startup pioneering the use of secure hardware with enclave-based solutions for trustworthy AI. This collaboration aims to demonstrate how AI governance policies can be enforced with cryptographic guarantees. In our first joint project, we created a proof-of-concept demonstration of confidential inference. We provide details of this work here because a crucial step to potential adoption of these mechanisms is demonstration that various use cases are practical using current technology.

Description of Project

Consider here two parties:

1. an AI custodian with a powerful AI model
2. an AI borrower who wants to run the model on their infrastructure but is not to be trusted with the weights directly

The AI custodian wants technical guarantees that:

1. the model weights are not directly accessible to the AI borrower.
2. trustable telemetry is provided to know how much computing is being done.
3. a non-removable off-switch button can be used to shut down inference if necessary.

Current AI deployment solutions, where the model is shipped on the AI borrower infrastructure, provide no IP protection, and it is trivial for the AI borrower to extract the weights without awareness from the custodian.

Through this collaboration, we have developed a framework for packaging and deploying models in an enclave using Intel secure hardware. This enables the AI custodian to lease a model, deployed on the infrastructure of the AI borrower, while the hardware guarantees that the weights are protected, and the trustable telemetry for consumption and off-switch are enforced. While this proof-of-concept is not necessarily deployable as is due to performance (we used Intel CPUs¹,) and specific hardware attacks that need mitigation, it serves as a demonstration of how secure enclaves can enable collaboration under agreed terms between parties with potentially misaligned interests.

By building upon this work, one can imagine how the US could lease its advanced AI models to allied countries while ensuring the model’s IP is protected and the ally’s data remains confidential and not exposed to the model provider. By developing and evaluating frameworks for hardware-backed AI governance, FLI and Mithril hope to encourage the creation and use of such technical measures so that we can keep AI safe without compromising the interests of AI providers, users, or regulators.

Future Projects Planned

Many other capabilities are possible, and we plan to rollout demos and analyses of more technical governance approaches in the coming months. The topic of BIS’s solicitation is one such approach: hardware could require remote approval if it identifies as part of a cluster satisfying some set of properties including size, interconnection throughput, and/or certificates of authorization. The objectives of the AC/S IFR could be further achieved through a secure training faculty, whereby an authority metes out authorized training compute cycles that are required for large training runs to be able to take place.² This secure training faculty could include a training monitor where all ML training runs above a threshold cluster size require, by law, licensing and compute training monitoring. In this process, licensing could be required via regulation requiring cluster limiting in all GPUs, and commitment to training monitoring could be required to obtain a license for training.

Many of these solutions can be implemented on existing and widely deployed hardware to allow AI compute governance to be backed by hardware measures. This addresses concerns that compute governance mechanisms are unenforceable or enforceable only with intrusive surveillance. The security of these measures needs testing and improvement for some scenarios, and we hope these demonstrations, and the utility of hardware-backed AI governance, will encourage both chip-makers and policymakers to include more and better versions of such security measures in upcoming hardware. Thus, while initially relying heavily on export restrictions and cooperation of data centers and cloud providers, eventually in principle on-chip mechanisms could carry the lion’s share of the responsibility for enforcement.

In this spirit, we recommend that: 

(a) BIS consider requiring the more robust secure enclaves on advanced ICs, rather than just TPMs, which can serve similar functions less robustly.

(b) BIS encourage and support engagement with chipmakers and other technical experts to audit, test, and improve security levels of hardware security measures.

We welcome more engagement and collaboration with BIS on this front.

---

General Comments

#1 – The Inclusion of Civil Society Groups in Input for BIS Rules

In its responses to comments to the IFR, BIS has made clear that the input of Technical Advisory Committees (TACs) is an important aspect of deliberating and instating new rules on export controls. It is also clear that the BIS annual report allows industry players to offer feedback on export control trade-offs for semiconductors, as outlined in ECRA Sections 1765 and 1752, and on national security issues under Sections 4812 and 4811. However, it’s not evident if civil society actors have the same opportunities for comment and input, aside from this specific Request for Comment. There is now a significant and diverse set of AI policy groups in the civil society ecosystem, populated – as in the case of FLI –  by some of the world’s leading experts from academia, government and industry. These actors possess a vital viewpoint to share on export control beyond the perspectives typically shared by industry. We invite BIS to clarify and make explicit the requirement for considerable input from civil society actors when it comes to the opportunities listed above, and those in the years to come.

#2 Clarifying Rule Applying to Those States which Facilitate Third Party WMD Activities

We commend the actions outlined within the AC/S IFR to ensure that export controls facilitate the restriction of weapons of mass destruction (WMD) related activities. FLI has published multiple reports on cyber, nuclear, chemical, and biological risks that intersect with the development of advanced AI systems. However, this risk does not emanate from state actors alone. In fact, several reports published over the last year demonstrate these same threats from non-state actors. We invite that BIS clarify that the IFR applies both to states listed in Country Group D:5 (and elsewhere) that use semiconductor technology for indigenous WMD-related activities, and those that are liable to share these technologies with allied and sponsored non-state actors, which in turn could use them for furthering WMD-activities.

#3 Preventing a Chilling Effect on Friendly US-China Cooperation

We support that BIS has clarified its position on § 744.6 in light of concerns that an overreach of the AC/S IFR might have a chilling effect on academic and corporate cooperation between Chinese and American persons and entities, cooperation with which may in fact forward the economic and national-security interests of the United States. We ask that the BIS expand on this acknowledgement by positively affirming in a separate section that such cooperation is welcome and within the remit of the AC/S IFR. The absence of clarity over this rule could detrimentally impact the current balance of US-China cooperation, threatening global stability and harming US national-security interests.

#4 On National Security Updates to the IFR

We welcome the BIS decision to introduce a density performance parameter to ensure that less powerful chips cannot be ‘daisy-chained’ into more powerful technologies and hence circumvent the principle purpose of the BIS rule. We also commend the use of a tiered approach when it comes to control of advanced integrated circuits. We hope that the BIS takes further account of emerging technological developments in hardware governance. For instance, new and innovative secure training hardware governance mechanisms (see point #5) can be required of IC makers in order to help prevent training of dual use models using unauthorized, heterogeneous distributed training.

#5 On addressing access to “development” at an infrastructure as a service (IaaS) provider by customers developing or intending to develop large dual-use AI foundation models with potential capabilities of concern, such as models exceeding certain thresholds of parameter count, training compute, and/or training data.

We welcome discussion on thresholds and potential capabilities of concern with regards to large dual-use foundation models. However, it is important to underscore that there should be explicit authority to change (and likely lower) these thresholds over time. This is because large dual-use AI foundation models with a certain set of thresholds held constant may become more powerful and dangerous to due to other factors.

For instance, algorithmic improvements in an AI model may significantly drive dual-use risk even if parameter count and training compute are held constant. In addition, the threshold of training data cannot just be quantitative but also qualitative – a model trained on higher quality or more dangerous (albeit smaller) training datasets can still present capabilities of concern.

Finally, the IFR would benefit from explicit discussion of the unique risk profile for capabilities of concern presented by dual-use AI models with widely available model weights . Models with widely available model weights at the same threshold as closed models will likely present greater potential capabilities of concern, as the guardrails from these models are more easily removed (if there are guardrails in the first place) and the models can be fine-tuned, using relatively little compute resources, to improve specific capabilities of concern.

Closing Remarks

We appreciate the thoughtful approach of BIS to the development of the AC/S IFR and are grateful for the opportunity to contribute to this important effort. We hope to continue engaging with this project and subsequent projects seeking to ensure AI does not jeopardize the continued safety, security, and wellbeing of the United States.

---

^{↩ 1} While we used CPUs in this case, a variation of this proof of concept would also work for GPUs, as they also support the Trust Platform Module (TPM) and secure enclave architectures.

^{↩ 2} This mechanism also facilitates various future auditability affordances.

Request for Information (CISA-2023-0027-0001) on “Shifting the Balance of Cybersecurity Risk: Principles and Approaches for Secure by Design Software”

Organization: Future of Life Institute

Point of Contact: Hamza Tariq Chaudhry, US Policy Specialist. hamza@futureoflife.blackfin.biz 

About the Organization

The Future of Life Institute (FLI) is an independent nonprofit organization with the goal of reducing large-scale risks and steering transformative technologies to benefit humanity, with a particular focus on artificial intelligence (AI). Since its founding, FLI has taken a leading role in advancing key disciplines such as AI governance, AI safety, and trustworthy and responsible AI, and is widely considered to be among the first civil society actors focused on these issues. FLI was responsible for convening the first major conference on AI safety in Puerto Rico in 2015, and for publishing the Asilomar AI principles, one of the earliest and most influential frameworks for the governance of artificial intelligence, in 2017. FLI is the UN Secretary General’s designated civil society organization for recommendations on the governance of AI and has played a central role in deliberations regarding the EU AI Act’s treatment of risks from AI. FLI has also worked actively within the United States on legislation and executive directives concerning AI. Members of our team have contributed extensive feedback to the development of the NIST AI Risk Management Framework, testified at Senate AI Insight Forums, participated in the UK AI Summit, and connected leading experts in the policy and technical domains to policymakers across the US government. We thank the Cybersecurity and Infrastructure Security Agency (CISA) for the opportunity to respond to this request for comments (RfC) regarding Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights, as specified in the White House Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.

---

Executive Summary

The Future of Life Institute (FLI) has a long-standing tradition of thought leadership on AI governance toward mitigating the risks and maximizing the benefits of AI. As part of this effort, we have undertaken research and policy work focused on

The principles outlined in CISA’s Secure by Design white paper offer a tractable foundation for ensuring the security of traditional software systems. However, as the RfI suggests, there are security considerations unique to AI that are not covered by, or necessitate reinterpretation of, these principles. Focusing on AI as software, we advocate for four core principles (Protect, Prevent, Strengthen, and Standardize) for actions taken by CISA when ensuring adherence by developers to secure by design principles: 

1. Protect advanced AI models developed in the United States from theft by malicious state and non-state actors, and from manipulation by these actors.
2. Prevent advanced AI systems from being used to launch AI-powered cyberattacks, both targeted at other kinds of software and also at the AI software itself.
3. Strengthen requirements that must be met before integrating advanced AI into cyber-defense systems, to ensure that cyber-defenses are not vulnerable to data poisoning, bias and other AI-derived harms.
4. Standardize ontologies and terminology unique to AI to inform the safe development, deployment, and governance of AI in the context of cybersecurity.

In line with these principles, we offer the following contributions:

1. Offer a Framework for a ‘Secure By Design‘ Technical Solution for AI systems. The RfI is clear that ‘AI is software and therefore should adhere to secure by design principles.’ Using advanced AI for formal verification and mechanistic interpretability and relying on prior innovations such as cryptographic guarantees, we offer a framework for ‘provably safe’ AI systems, providing necessary conditions to make them secure by design.
2. Analysis of, and Recommendations to Mitigate, Harms Posed to Software at the Intersection of AI and Cybersecurity. The white paper extensively discusses the complexity of guarding against and responding to software vulnerabilities, and the RfI poses several questions regarding these issues. As advancements in AI have accelerated, the cyber threats posed to software underpinning our digital and physical infrastructure have also increased. In our policy contribution to this effort, we offer analysis of the risks posed to software by AI-cyber threats, alongside recommendations to mitigate them to protect and strengthen software security. This includes recommendations for the national cybersecurity strategy and guidance for the integration of AI in national security systems. 
3. Ensuring Transparency and Accountability from AI Products: In keeping with a fundamental principle of the Secure by Design framework, which directs developers of software – including AI – to develop ‘safe and secure products’, we offer recommendations to ensure that any development of advanced AI software is transparent and that developers are held accountable for the advanced AI they produce. This includes suggestions for licensing, auditing, and assigning liability for resulting harms.
4. Foster the Development of Common Ontologies and Terminology: In order for software systems to be safe-by-design, they must be verifiable against technical specifications. However, these technical specifications and their expression through common ontologies have yet to be standardized. We recommend that CISA support the standardization of these ontologies and terms.

---

1. Technical Framework for ‘Secure By Design’ AI systems

Background – Summary of Research Findings

In September 2023, FLI founder and President Dr. Max Tegmark published a paper on provably safe AI systems, in co-authorship with AI safety pioneer Dr. Steve Omohundro.¹ Here, we condense the findings of that paper into a secure by design technical framework for AI systems.

The paper proposes a technical solution to designing secure AI systems by advancing the concept of provably safe AI systems. This framework has five components:

1. A Provably Compliant System (PCS) is a system (hardware, software, social or any combination thereof) that provably meets certain formal specifications.
2. Proof-carrying code (PCC) is software that is not only provably compliant, but also carries within it a formal mathematical proof of its compliance, i.e., that executing it will satisfy certain formal specifications. Because of the dramatic improvements in hardware and machine learning (ML), it is now feasible to expand the scope of PCC far beyond its original applications such as type safety, since ML can discover proofs too complex for humans to create.
3. Provably Compliant Hardware (PCH) is physical hardware the operation of which is governed by a Provable Contract.
4. Provable Contracts (PC) govern physical actions by using secure hardware to provably check compliance with a formal specification before actions are taken. They are a generalization of blockchain ”Smart Contracts” which use the cryptographic guarantees to ensure that specified code is correctly executed to enable blockchain transactions. Provable contracts can control the operation of devices such as drones, robots, GPUs and manufacturing centers. They can ensure safety by checking cryptographic signatures, zero-knowledge proofs, proof-carrying code proofs, etc. for compliance with the specified rules.
5. Provable Meta-Contracts (PMC) impose formal constraints on the creation or modification of other provable contracts. For example, they might precisely define a voting procedure for updating a contract. Or they might encode requirements that provable contracts obey local laws. At the highest level, a PMC might encode basic human values that all PCs must satisfy.

Taking these components together, provably compliant systems form a natural hierarchy of software and hardware. If a GPU is PCH, then it should be unable to run anything but PCC meeting the GPU’s specifications. As far as software is concerned, PCH guarantees are analogous to immutable laws of physics: the hardware simply cannot run non-compliant code. Moreover, a PCC can be often be conveniently factored into a hierarchy of packages, subroutines and functions that have their own compliance proofs. If a provable contract controls the hardware that PCC attempts to run on, it must comply with the specification. Compliance is guaranteed not by fear of sanctions from a court, but because it is provably physically impossible for the system to violate the contract.

Implications for Secure by Design AI

Due to the black box nature of AI systems, some AI experts argue that it is nearly impossible to fully secure an AI system through technical means alone.²³

By applying and building on the research of Dr. Tegmark and Dr. Omohundro, however, developers can build technical components into AI systems that create a pathway to verifiably secure systems. Hence, this line of research serves as proof of concept that securing AI systems by design is technically feasible. Coupled with thoughtful policy mechanisms to strengthen the security of AI systems, we believe this type of technical solution can be effective in ensuring secure by design AI systems. We look forward to engaging with CISA in the future to expand this research project and integrate it with ‘secure by design’ guidance offered by CISA to AI software developers.

---

2. Problem Analysis and Recommendations to Mitigate Harms Posed to Software at the Intersection of AI and Cybersecurity

Numerous reports have pointed to the ways that AI systems can make it easier for malevolent actors to develop more virulent and disruptive malware, and can lower the barrier of technical expertise necessary for motivated individuals to carry out cyberattacks.⁴⁵ AI systems can also help adversaries automate attacks on cyberspaces, increasing the efficiency, creativity and impact of cyberattacks via novel zero-day exploits (i.e. previously unidentified vulnerabilities), targeting critical infrastructure, better automating penetration scans and exploits, and enhancing techniques such as phishing and ransomware. As AI systems are increasingly empowered to plan and execute self-selected tasks to achieve assigned objectives, we can also expect to see the emergence of autonomous hacking activities initiated by these systems in the near future. All of these developments have changed the threat landscape for software vulnerabilities. This policy contribution first summarizes these threats, and then provides recommendations that could help companies, government entities and other actors protect their software.

Threat Analysis

1. Threat to Software Underpinning Critical Infrastructure. An increasing proportion of US critical infrastructure, including those pieces relevant to health (e.g. hospital systems), utilities (e.g. heating, electrical supply and water supply), telecommunications, finance, and defense are now ‘on the grid’ – reliant on integrated online software- leaving them vulnerable to potential cyberattacks by malicious actors. Such an attack could, for instance, shut off the power supply of entire cities, access high-value confidential financial or security information, or disable telecommunications networks. AI systems are increasingly demonstrating success in exploiting such vulnerabilities in the software underpinning critical infrastructure.⁶ Crucially, the barrier to entry, i.e. the level of skill necessary, for conducting such an attack is considerably lower with AI than without it, increasing threats from non-state actors and the number and breadth of possible attempts that may occur. Patching these vulnerabilities once they have been exploited takes time, which means that painful and lasting damage may be inflicted before the problem is remedied.
2. Cyber-vulnerabilities in Labs Developing Advanced AI Software. As the RfI outlines, there is a need to ensure that AI is protected from vulnerabilities just as is the case with traditional software. The “Secure by Design” white paper advocates for software developers to “take ownership of their customer’s security outcomes.” This responsibility should also apply to AI developers, compelling them to address AI-specific cyber vulnerabilities that affect both product safety for customers and wider societal concerns. The most advanced AI systems in the world – primarily being developed in the United States – are very likely to be targeted by malicious state and non-state actors to access vital design information (e.g., the model weights underpinning the most advanced large language models). Because developing these systems is resource intensive and technically complex, strategic competitors and adversaries may instead steal these technologies without taking the considerable effort to innovate and develop them, damaging U.S. competitiveness and exacerbating risks from malicious use. Once model weights are obtained, these actors could relatively easily remove the safeguards from these powerful models, which normally protect against access to dangerous information such as how to develop WMDs. Several top cybersecurity experts have expressed concerns that the top AI labs are ill-equipped to protect these critical technologies from cyber-attacks.
3. Integration of Opaque, Unpredictable and Unreliable AI-Enabled Cybersecurity Systems. Partly to guard against exploitation of vulnerabilities, there has been increasing interest in the potential use of AI systems to enhance cybersecurity and cyber-defense. This comes with its own set of threats, especially with opaque AI systems for which behavior is extremely difficult to predict and explain. Data poisoning – cases where attackers manipulate the data being used to train cyber-AI systems – could lead to systems yielding false positives, failing to detect intrusions, or behaving in unexpected, undesired ways. In addition, the model weights of the systems themselves can be largely inferred or stolen using querying techniques designed to find loopholes in the model. These systems could also autonomously escalate or counter-attack beyond their operators’ intentions, targeting allied systems or risking serious escalations with adversaries.

In summary, software vulnerabilities are under greater threat of covert identification and exploitation due to AI-powered cyberattacks. At the same time, integration of AI into cybersecurity systems to guard software presents unique threats of its own. Finally, the state of the art AI software being developed within leading labs within the United States is itself under threat from malicious actors.

Recommendations for Threat Mitigation

To mitigate these problems, we propose the following recommendations:

1. Industry and governmental guidance should focus explicitly on AI-enabled cyber attacks in national cyber strategies: AI goes completely unmentioned in the National Cybersecurity Strategy Implementation Plan published by the White House in July 2023, despite recognition of cyber risks of AI in the National Cybersecurity Strategy itself. AI risks need to be integrated explicitly into a broader cybersecurity posture, including in the DOD Cyber Strategy, the National Cyber Incident Response Plan (NCIRP), the National Cybersecurity Investigative Joint Task Force (NCIJTF) and other relevant plans.
2. Promulgate Guidance for Minimum Standards for Integration of AI into Cybersecurity Systems and Critical Infrastructure: Integrating unpredictable and vulnerable AI systems into critical cybersecurity systems may create cyber-vulnerabilities of its own. Minimum standards regarding transparency, predictability and robustness of these systems should be set up before they are used for cybersecurity functions in critical industries. Additionally, building on guidance issued in accordance with EO 13636 on Improving Critical Infrastructure Cybersecurity4, EO 13800 on Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure5, and the Framework for Improving Critical Infrastructure Cybersecurity published by NIST6, AI-conscious standards for cybersecurity in critical infrastructure should be developed and enforced. Such binding standards should account in particular for risks from AI-enabled cyber-attacks, and should be developed in coordination with CISA, SRMA and SLTT offices.

---

3. Ensuring Transparency and Accountability from AI Products

We ask that CISA and DHS consider the following recommendations to guarantee the transparent and accountable development of secure AI. In addition, these recommendations would ensure that developers take responsibility for software security and do not impose unfair costs on consumers, a fundamental principle of the Secure by Design framework. To protect and strengthen AI systems, we recommend that CISA:

1. Require Advanced AI Developers to Register Large Training Runs and to “Know Their Customers”: The Federal Government lacks a mechanism for tracking the development and proliferation of advanced AI systems, despite there being a clear need expressed by agencies including CISA to guarantee security of AI software. In addition, these advanced AI systems could exacerbate cyber-risk for other kinds of software. In order to mitigate cybersecurity risks, it is essential to know what systems are being developed and what kinds of actors have access to them. Requiring registration for the acquisition of large amounts of computational resources for training advanced AI systems, and for carrying out the training runs themselves, would help with tracking and evaluating possible risks and taking appropriate precautions. “Know Your Customer” requirements, similar to those imposed in the financial services industry, would reduce the risk of powerful AI systems falling into the hands of malicious actors.
2. Establish, or Support the Establishment of, a Robust Pre-deployment Auditing and Licensure Regime for Advanced AI Systems: In order to ensure the security of AI software, it must first be guaranteed that AI systems do not behave in dangerous and unpredictable ways. Advanced AI that can pose risks to cybersecurity, may be integrated into a system’s critical functions, or may be misused for malicious attacks are not presently required to undergo independent assessment for safety, security, and reliability before being deployed. Additionally, there are presently no comprehensive risk assessments for AI systems across their extensive applications and integrations. Requiring licensure before potentially dangerous advanced AI systems are deployed, contingent on credible independent audits for compliance with minimum standards for safety, security, and reliability, would identify and mitigate risks before the systems are released and become more difficult to contain. Audits should include red-teaming to identify cyber-vulnerabilities and to ensure that systems cannot be readily used or modified to threaten cybersecurity.
3. Clarify Liability for Developers of AI Systems Used in Cyber-attacks: In order to encourage transparency, accountability and generally protect software from AI-powered cyberattacks, it is critical to establish a liability framework for developers of AI systems that could conceivably be used to exploit cyber-vulnerabilities. At present, it is not clear under existing law whether the developers of AI systems used to, e.g., damage or unlawfully access critical infrastructure would be held liable for resulting harms. Absolving developers of liability in these circumstances creates little incentive for profit-driven developers to expend financial resources on precautionary design principles and robust assessment. Because these systems are opaque and can possess unanticipated, emergent capabilities, there is inherent risk in developing systems expected to be used in critical contexts as well as advanced AI systems more generally. Implementing strict liability when these systems facilitate or cause harm would better incentivize developers to take appropriate precautions against cybersecurity vulnerabilities, critical failure, and the risk of use in cyber-attacks.

---

4. Foster the Development of Common Ontologies and Terminology

The lack of standardized ontologies, terminology, and comprehensive risk management frameworks complicates the security landscape for AI systems, which present novel and amplified challenges compared to traditional software.⁷ In order for software systems to be safe by design, they must be verifiably compliant with technical specifications, and technical specifications are expressed using ontologies, i.e. graphical schema representing the entity types, properties, relationships, and constraints within one or more domains of knowledge. Furthermore, the general purpose nature of many machine learning systems, which inherently have a wide range of applications, renders the assessment of their risks particularly challenging. To standardize these shared approaches we recommend that CISA:

1. Induce and support the development of shared ontologies at the intersection of AI and cybersecurity⁸: These should be developed within and across industries, government, and nations so that broader and deeper networks of compatible and provable security can more easily flourish. Likewise, development of crosswalks, bridge ontologies, and ontology alignment faculties would also aid such an ecosystem.⁹ 
2. Support the standardization of terminology relevant to AI and cybersecurity: AI security approaches have often borrowed terms, frameworks, and techniques from related fields like cybersecurity, hardware, and system safety engineering.¹⁰ While this can occasionally be appropriate, it often leads to misinterpretations that prevent the effective use of established risk mitigation strategies. Formal definitions for what constitutes, e.g., audits, system requirements and safety requirements should be established within the context of AI and cybersecurity to avoid conflation with other fields and inform downstream management.¹¹

---

Closing Remarks

We appreciate the thoughtful approach of CISA to the development of the Secure by Design Software framework and are grateful for the opportunity to contribute to this important effort. We hope to continue engaging with this project and subsequent projects seeking to ensure AI software does not jeopardize the continued safety, security, and wellbeing of the United States.

---

^{↩ 1} Max Tegmark and Steve Omohudro. (2023). Provably safe systems: the only path to controllable AGI. arXiv preprint arXiv:2309.01933.

^{↩ 2} Mike Crapps. (March, 2023). Making AI trustworthy: Can we overcome black-box hallucinations? TechCrunch.

^{↩ 3} W.J. von Eschenbach. (2021). Transparency and the black box problem: Why we do not trust AI. Philosophy & Technology, 34(4), 1607-1622.

^{↩ 4} Bécue, A., Praça, I., & Gama, J. (2021). Artificial intelligence, cyber-threats and Industry 4.0: Challenges and opportunities. Artificial Intelligence Review, 54(5), 3849-3886.

^{↩ 5} Menn, J. (May, 2023). Cybersecurity faces a challenge from artificial intelligence’s rise. Washington Post.

^{↩ 6} Office of Intelligence and Analysis. Homeland Threat Assessment 2024. Department of Homeland Security.

^{↩ 7} While the NIST AI RMF may constitute a standardized RMF, we believe it still requires considerable iteration to fill gaps in AI risk management.

^{↩ 8} A shared ontology – or a shared schematic representation of concepts and terminologies across different contexts – is often developed to help collaborate on workflows. For instance, a shared biomedical ontology could help computer systems and decision-makers collate and analyze information across several different biomedical websites. In this context, it would help different actors working with a wide variety of systems in diverse contexts to effectively cooperate on AI and cybersecurity issues.

^{↩ 9} Crosswalks effectively function as translators in cases where complex networks of systems and data employ different terminologies and classifications for concepts. Crosswalks provide mappings to allow translation between these different schemes. A bridge ontology can serve a similar function, representing the construction of a bridge between different ontologies. All of these efforts feed into ontology alignment, the practice of ensuring correspondence between concepts in different ontologies.

^{↩ 10} Heidy Khlaf. (March, 2023). Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems. Trail of Bits.

^{↩ 11} We commend current efforts in this regard such as the NIST glossary of terms as a starting point. (See Trustworthy and Responsible AI Resource Center. Glossary. NIST) We request that this glossary be expanded and more widely adopted and applied to serve the function of effectively standardizing terminology. CISA can play a critical role here by incorporating interpretations of cybersecurity terms in AI contexts where their meaning may be more ambiguous due to distinctions between AI and traditional software.

This letter was published in English but is also available in Spanish (Español), French (Français), German (Deutsch), Portuguese (Português), Arabic (العربية), and Chinese (中文).

Our world is in grave danger. We face a set of threats that put all humanity at risk. Our leaders are not responding with the wisdom and urgency required.   

The impact of these threats is already being seen: a rapidly changing climate, a pandemic that killed millions and cost trillions, wars in which the use of nuclear weapons has been openly raised. 

There could be worse to come. Some of these threats jeopardise the very existence of life on earth. We do not yet know how significant the emerging risks associated with Artificial Intelligence will be.

We are at a precipice. 

The signatories of this letter call on world leaders to work together to address these existential threats more decisively. We welcome people of all communities, generations, and political views to join us in asking for courageous decision-making – for the sake of our common future. 

The knowledge and resources to address these challenges exist. But too many of our leaders lack the political will or capability to take decisive action. They seek short-term fixes over long-term solutions.  

In a year when half the world’s adult population face elections, we urge all those seeking office to take a bold new approach. We need long-view leadership from decision-makers who understand the urgency of the existential threats we face, and believe in our ability to overcome them. 

Long-view leadership means showing the determination to resolve intractable problems not just manage them, the wisdom to make decisions based on scientific evidence and reason, and the humility to listen to all those affected. Long-view leaders must have the moral strength to address both current concerns and long-term risks, often at the expense of vested interests.  

Such values should be common to all political leaders. But they are woefully missing in so many. We need leaders, women and men, who consistently demonstrate the courage to:  

1. Think beyond short-term political cycles and deliver solutions for both current and future generations.  
2. Recognise that enduring answers require compromise and collaboration for the good of the whole world.  
3. Show compassion for all people, designing sustainable policies which respect that everyone is born free and equal in dignity and rights. 
4. Uphold the international rule of law and accept that durable agreements require transparency and accountability.   
5. Commit to a vision of hope in humanity’s shared future, not play to its divided past. 

These principles of long-view leadership can inform urgent changes in policy. Governments can get to work now to agree how to finance the transition to a safe and healthy future powered by clean energy, relaunch arms control talks to reduce the risk of nuclear war, save millions of lives by concluding an equitable pandemic treaty, and start to build the global governance needed to make AI a force for good, not a runaway risk. 

As leaders prepare to gather in New York in September for the UN Summit of the Future, it is time to change direction. The biggest risks facing us cannot be tackled by any country acting alone. Yet when nations work together, these challenges can all be addressed, for the good of us all. 

Despite the seriousness of these existential threats, hope remains. Our best future can still lie ahead of us. We call on leaders to take the long view, and show the courage to lead us to that better future.

I. Background on FLI

The Future of Life Institute (FLI) is an independent non-profit that works to steer transformative technology towards benefiting life and away from extreme large-scale risks. We work through policy advocacy at the UN, in the EU and the US, and have a long history of grants programmes supporting such work as AI existential safety research and investigations into the humanitarian impacts of nuclear war. This current request for proposals is part of FLI’s Futures program, which aims to guide humanity towards the beneficial outcomes made possible by transformative technologies. The program seeks to engage a diverse group of stakeholders from different professions, communities, and regions to shape our shared future together.

II. Request for Proposal

Call for proposed designs for global institutions governing AI

FLI is calling for research proposals with the aim of designing trustworthy global governance mechanisms or institutions that can help stabilise a future with 0, 1, or more AGI projects. These proposals should outline the specifications needed to reach or preserve a secure world, taking into account the myriad threats posed by advanced AI. Moreover, we expect proposals to specify and justify whether global stability is achieved by banning the creation of all AGIs, enabling just one AGI system and using it to prevent the creation of more, or creating several systems to improve global stability. There is the option for proposals to focus on a mechanism not dependent on a particular scenario, or one flexible enough to adapt to 0, 1 or more AGI projects. Nonetheless, it is recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued. In that sense, as well as pitching a particular control mechanism, each proposal is also making a case for how humanity is kept safe in a specific future scenario.

FLI’s rationale for launching this request for proposal

Reaching a future stable state may require restricting AI development such that the world has a.) no AGI projects; b.) a single, global AGI project, or c.) multiple AGI projects. By AGI, we refer to Shane Legg’s definition of ‘ASI’: a system which outperforms 100% of humans at a wide range of non-physical tasks, including metacognitive abilities like learning new skills (see grid below, taken from Legg’s paper on this). A stable state would be a scenario that evolves at the cautious timescale determined by thorough risk assessments rather than corporate competition.

a. No AGI projects

As recently argued by the Future of Life Institute’s Executive Director, Anthony Aguirre, if there is no way to make AGI – or superhuman general-purpose AI – safe, loyal, and beneficial, then we should not go ahead with it. In practice, that means that until we have a way of proving that an AGI project will not, upon completion or release, take control away from humanity or cause a catastrophe, AGI models should be prevented from being run. Equally, harnessing the benefits and preventing the risks of narrow AI systems and controllable general purpose models still requires substantial global cooperation and institutional heft, none of which currently exists. Proposals focusing on such initiatives might include mechanisms for:

• Risk analysis;

• Predicting, estimating, or evaluating the additive benefit of continuing AI capability development;

• Ensuring no parties are cheating on AGI capability development;

• Building trust in a global AI governance entity; or,

• Leveraging the power of systems to solve bespoke problems that can yield societal and economic benefits.

b. One AGI project

As Aguirre points out, even if an AGI is ‘somehow, made both perfectly loyal/subservient to, and a perfect delegate for, some operator’, that operator will rapidly acquire far too much power – certainly too much for the comfort of existing powers. He notes that other power structures ‘will correctly think… that this is an existential threat to their existence as a power structure – and perhaps even their existence period (given that they may not be assured that the system is in fact under control.)’ They may try to destroy such a capability. To avoid this imbalance or its disastrous results, the single AGI will either need to be created by a pre-existing cooperation of the great powers, or brought under the control of a new global institution right after it is produced. Again, the world lacks these institutions, and forming them will require extensive research and thought. Alongside variations on the ideas listed above, proposals for such establishments and connected questions needing answers could also include:

• Mechanisms for distributing the benefits of the centralised capabilities development efforts: how is this done from a technical perspective? Is it all of humanity? Only the signatories of an international treaty? How are economic benefits shared and is that the same or different than new knowledge?

• Mechanisms for preventing authoritarian control? How can power be centralised without corruption?

• Intellectual property structures that would enable and incentivize commercialization of breakthroughs discovered in an AGI system.

• Mechanisms for determining if a new discovery should be developed by a private entity or if it is for the common good (e.g. radical climate intervention)

• Potential governance structures for such an entity: who makes the decisions about development? Risks? How is the concentration of power kept in check and accountable to the globe? How is capture by special interests, spies, or geopolitical blocs prevented?

• Verification mechanisms to ensure no one is cheating on AGI capability development-How to prevent others from cheating? Penalties? Surveillance mechanisms? How much does this need to vary by jurisdiction?

• How is such an organization physically distributed?

• Mechanisms for ensuring the security of such an entity, preventing leaks and accidents.

• Can realpolitik support centralised global AGI development? What would it take to actually convince the major states that they wouldn’t effectively be giving up too much sovereignty or too much strategic positioning?

c. Multiple AGI projects

Similar concerns arise if multiple AGI systems emerge. A delicate balance of power must be strenuously maintained. Furthermore, with all of these scenarios there will ensue the significant dual problem of on the one hand limiting the associated risks of such powerful AI systems, and, on the other, distributing the associated benefits – of which we can expect many. Some problems along those lines in need of solutions:

• How much power should an assembly of represented states have vs. bureaucratic managers vs. scientists?

• What is an equitable, fair, and widely agreeable distribution of voting power among the represented states? Should simple- or super-majorities be required? Should Vanuatu have the same weight as Japan?

• Who gets to decide how or whether AI should influence human values? What and whose values get enshrined in a new global institution for governing AI? And if no one’s in particular, what mechanisms can help to maintain room for personal convictions, free thinking, community traditions such as religion, careful decision-making and contradicting values? (see Robert Lempert’s new RAND paper)

d. Flexible to different numbers of AGI projects

Dividing scenarios into the three above groups will hopefully yield a balanced sample of each outcome. Equally, urging primary investigators to select just one of these categories may help to encourage concrete scenarios for what a well-managed future with advanced AI could look like – in short, push them to pick only their preferred future. However, some applicants may wish to submit proposals for mechanisms flexible enough to adjust to varying numbers of AGI systems. These applicants will need to address how their mechanism can ensure stability in a world where the number of AGI systems can keep changing. Alternatively, there may be mechanisms or institutions proposed whose function is not dependent on a particular number of AGI projects. In such cases, it is still recommended that applicants consider in which scenario their proposed mechanism would best perform, or be most valued; nonetheless, we leave this option available for those cases where such a consideration proves to be unenlightening.

The success of all of these case groups depends upon a diligent and consistent consideration of the risks and benefits of AI capability development. Any increase in the power of intelligent systems must proceed in accordance with an agreed acceptable risk profile, as is done in the development of new drugs. However, it is not clear what the structure of such an organization would look like, how it would command trust, how it would evade capture, or how it could endure as a stable state. This request for proposals can be summarised as a search for this clarity. A better, more informed sense of where we wish to be in a few years, of what institutions will best place us to tackle the upcoming challenges, will be invaluable for policymakers today. Such a well-researched north star can help the world to reverse engineer and work out what governments should be doing now to take humanity in a better, safer direction.

Without a clear articulation of how trustworthy global governance could work, the default narrative is that it is impossible. This RFP is thus borne both of an assessment of the risks we face, and of sincere hope that the default narrative is wrong, a hope that if we keep it under control and use it well, AI will empower – rather than disempower – humans the world over.

Existing projects

FLI is by no means creating a new field here. Promising initiatives already in the works, which may inspire researchers applying to this program, include the following:

• FLI’s international AI governance organization proposal

• Davidad’s Open Agency Architecture

• The Oxford Martin AI Governance Initiative’s whitepaper: International Governance of Civilian AI: A Jurisdictional Certification Approach

• Conjecture’s Multinational AGI Consortium (MAGIC) – a ‘CERN for AI’

III. Evaluation Criteria & Project Eligibility

Proposals will be evaluated according to the track record of the researcher, the proposal’s originality or potential to be transformative, the potential for the proposed activity to advance knowledge of the coordination problems for mitigating AGI risk, and how convincingly the proposal accounts for the range of AGI risks.

Grants applications will be subject to a competitive process of external and confidential peer review. We intend to support several proposals. Accepted proposals will receive a one-time grant of $15,000, to be used at the researcher’s discretion. Grants will be made to nonprofit organizations, with institutional overhead or indirect costs not exceeding 15%.

IV. Application process

All applications should be submitted electronically through this form. We will accept applications internationally. But all applicants should have a nonprofit organization with which they are associated to accept the funding. We will not make grants directly to individuals.

Applications deadline: 1st April 2024.

External reviewers invited by FLI will then evaluate all the proposals according to the above criteria, and decisions will be shared by mid to late May. Completed research papers are due by 13th September.

All questions should be sent to grants@futureoflife.blackfin.biz.

I. Background on FLI

The Future of Life Institute (FLI) is an independent non-profit that works to steer transformative technology towards benefiting life and away from extreme large-scale risks. We work through policy advocacy at the UN, in the EU and the US, and have a long history of grants programmes supporting such work as AI existential safety research and investigations into the humanitarian impacts of nuclear war. This current request for proposals is part of FLI’s Futures program, which aims to guide humanity towards the beneficial outcomes made possible by transformative technologies. The program seeks to engage a diverse group of stakeholders from different professions, communities, and regions to shape our shared future together.

II. Request for Proposal

Call for proposals evaluating the impact of AI on Poverty, Health, Energy and Climate SDGs

The Future of Life Institute is calling for proposals for research evaluating in detail how artificial intelligence (AI) has so far impacted the Sustainable Development Goals (SDGs) relating to poverty, healthcare, energy and climate change, and how it can be expected to impact them in the near future. This research can examine either cases where AI is intended to address respective SDGs directly, or where AI has affected the realisation of these goals by its side effects. Each paper should select one SDG or target, analyse the impact of AI on its realisation up to the present, and explore the ways in which AI could accelerate, inhibit, or prove irrelevant to, the achievement of that goal by 2030. We acknowledge that AI is a broad term, encompassing systems that are both narrow and general with varying degrees of capability. Hence, for the purposes of this RFP we encourage using this taxonomy as a guide for exploring and categorising AI’s current and future uses.

FLI’s rationale for launching this request for proposal

Need for more detail on how AI can improve lives

There has been extensive academic research and, more recently, public discourse on the risks of AI. Experts have exposed the current harms of AI systems, as well as how increasing the power of these systems will scale these harms and even facilitate existential threats.

By contrast, the discussion around the benefits of AI has been quite ambiguous. The prospect of enormous benefits down the road from AI – that it will “eliminate poverty,” “cure diseases” or “solve climate change” – helps to drive a corporate race to build ever more powerful systems. But while it is clear that AI will make significant contributions to all of these domains, the level of capabilities necessary to realize those benefits is less clear.

As we take on increasing levels of risk in the race to develop more and more capable systems, we need a concrete and evidence-based understanding of the benefits, in order to develop, deploy and regulate this technology in a way that brings genuine benefits to people’s lives, all over the world. This understanding has real world impacts. For instance, if current AI models are already sufficient to solve major problems and meet global needs, then the way forward looks much more like applying and adapting what we have to the tasks at hand. As Future of Life Institute Executive Director Anthony Aguirre put it in a recent paper, ‘systems of GPT-4’s generation are already very powerful, and we have really only scratched the surface of what can be done with them.’

When considering the kinds of AI models we might need to achieve the SDGs in the near future, a recent paper provides a useful framework (see below) that grades AI models – those we already have and those not yet achieved – by generality and performance capability. For more on this, read the full paper.

Equally, if it becomes clear that the hurdles impeding the improvement of human lives are borne not of technological shortcomings but coordination problems, or sociological puzzles, then that too will have implications for the dispersal of future funding. The question then becomes: how can we know if AI is presently bringing, or able to bring, real benefits?

The SDGs

The Sustainable Development Goals (SDGs) remain the most broadly supported repository of high-priority problems for the world to solve, especially with regards to poverty, health, energy and climate-related challenges. The centrepiece of the 2030 Agenda for Sustainable Development adopted by all United Nations Member States in 2015, the 17 SDGs constitute an ambitious hope for a better world, but also, for our purposes, a set of concrete measurable targets against which to assess the extent of progress in the four defined areas. For clarity, the goals directly relevant to those focuses are 1 (Poverty), 3 (Health), 7 (Energy) and 13 (Climate).

The goals are interconnected. Solving one may involve or assist the solving of another. According to the UN, the goals “recognize that ending poverty and other deprivations must go hand-in-hand with strategies that improve health and education, reduce inequality, and spur economic growth – all while tackling climate change and working to preserve our oceans and forests.” Indeed, a 2020 paper written by Vinuessa et al. analysed the effect AI could have on all of the goals. It concluded that while AI could have both positive and negative impacts on the SDGs, the net effect of AI would be positive.

Nonetheless, each of them individually poses a formidable challenge, with its own specific contingencies and obstacles. Recent assessments of the state of progress on the SDGs painted a bleak picture. Many of them look unlikely to be achieved by the end of the decade. For instance, goals concerning hunger, malaria, employment, slum-dwelling proportions, greenhouse gas emissions and the extinction of threatened species are all deemed to be in the red by the UN, in part because of the indirect effects of COVID-19 – especially when it comes to poverty eradication (SDG 1). Evaluating the impact of AI on just one of these domains will be more than sufficient a task for a single research paper of approximately ten pages.

As the 2020 paper showed, there is cause for optimism about how AI might affect the achievement of each goal. But it is time we moved beyond the hypothetical, and ascertained the impact AI is already having on the pursuit of these targets. Only then can we proceed to assess what kinds of AI development will help to bring about the better world promised in the 2030 Agenda, and how we might pursue them.

Filling a gap

As noted in the overview analysis by Vinuesa et al, “self-interest can be expected to bias the AI research community and industry towards publishing positive results.” As a result, we lack objective, independent analysis of the impact of AI thus far. Given that AI is rapidly being integrated into all aspects of society, this gap in the research community now needs filling.

Sample proposal titles

These samples are to get researchers thinking about various approaches. The selection of SDGs does not imply a preference for those particular goals in the research proposed.

SDG 1

• How has AI been affecting the implementation of social support systems?
• What data do we have to suggest how AI will impact the goal of reducing poverty by half in 2030?
• What is the risk that general-purpose AI will significantly increase poverty by then?

SDG 3

• How has AI affected the goal of decreasing maternal mortality?

III. Evaluation Criteria & Project Eligibility

Proposals will be evaluated according to the track record of the researcher, the quality of the evaluation outline, the likelihood of the research yielding valuable findings, and the rigour of the proposed projection method.

Grants applications will be subject to a competitive process of external and confidential peer review. We intend to support several proposals. Accepted proposals will receive a one-time grant of $15,000, to be used at the researcher’s discretion. Grants will be made to nonprofit organizations, with institutional overhead or indirect costs not exceeding 15%.

IV. Application process

All applications should be submitted electronically through this form. We will accept applications internationally. But all applicants should have a nonprofit organization with which they are associated to accept the funding. We will not make grants directly to individuals.

Applications deadline: 1st April 2024.

External reviewers invited by FLI will then evaluate all the proposals according to the above criteria, and decisions will be shared by mid to late May. Completed research papers are due by 13th September.

All questions should be sent to grants@futureoflife.blackfin.biz. 

Our Futures Program, launched in 2023, aims to guide humanity towards the beneficial outcomes made possible by transformative technologies. This year, as part of that program, we are opening two new funding opportunities to support research into the ways that artificial intelligence can be harnessed safely to make the world a better place.

The first request for proposals (RFP) calls for papers evaluating and predicting the impact of AI on the achievement of the UN Sustainable Development Goals (SDGs) relating to poverty, healthcare, energy and climate change. The second RFP calls for designs of trustworthy global mechanisms or institutions to govern advanced AI in the near future.

Selected proposals in either category will receive a one-time grant of $15,000, to be used at the researcher’s discretion. We intend to make several grants in each track.

Applications for both tracks are now open and will remain so until April 1st, 2024.

The Windfall Trust aims to alleviate the economic impact of AI-driven joblessness by building a global, universally accessible social safety net

Image: Adapted from Scott Santens, CC BY-SA 2.0, via Wikimedia Commons

 

Advancing AI capabilities and the economy

As artificial intelligence progresses toward human-level capabilities, many of today’s AI companies’ goals are to create AI that is better than humans at all tasks, including all economically valuable work—something widely referred to as artificial general intelligence (AGI). 

Whether this vision appears achievable or not, many of the smartest engineers and experts, backed by some of the most well resourced companies in human history, are aggressively pursuing this goal—and are explicit about it. If they succeed in developing human level Artificial Intelligence, it’s likely that we’ll see widespread and unprecedented joblessness within our lifetime, raising critical questions about economic stability and the future of human employment. 

Preparing for a new economic paradigm

Governments and regulators may have tools, such as reskilling programs and temporary safety nets, to address short-term national economic challenges of job displacement. But in a world where AI rapidly results in widespread joblessness with limited new job creation, the conventional tools are likely to be insufficient. We must develop new tools and ideas to cushion this economic disruption and protect workers. 

Given the global and highly interconnected world economy, AI-induced joblessness will inevitably transcend borders. If advanced economies are not equipped to deal with AI’s impacts, how will governments lacking robust tax bases or social welfare systems manage the economic fallout of widespread rapid job displacement? The global scale and complexity of this challenge demand solutions that address the financial consequences of job loss on a scale and of a scope that meets the immensity of the challenge.

Recognizing these challenges, we believe it is imperative to start preparing for the kinds of transformative shifts that could break existing paradigms and fundamentally reshape our economies and societies. 

A new economic institution design for the age of AI

The Windfall Clause is a proposed mechanism wherein AI companies legally pre-commit to broadly investing profits into cushioning the economically disruptive impacts of AI. This clause is only triggered if their profits exceed a high predetermined threshold, such as a single company generating 1% of the world’s GDP (presently, even the most profitable company in the world is several hundreds of billions of dollars away from this threshold). But how would such profits be reinvested into society? What if there existed a fund that belonged to all of humanity—where everyone born on earth would be entitled to a share in its corpus? Could we seed such a fund with a small portion of the enormous wealth generated by AI to create a robust model for universal basic wealth?

The Windfall Trust aspires to research and create such an entity. The Future of Life Institute has been researching a roadmap for a fully fleshed-out design for such a trust, encompassing legal and financial structures, investment principles, payout algorithms, governance frameworks, and funding plans. Its overarching goal is to foster economic stability by distributing income from the trust, starting with the most economically vulnerable, to establish a universal human income floor.

Nothing of this scale has been attempted before, and there are many uncertainties in bringing the Windfall Trust to life. However, we are committed to building an economic institution to meet this challenge and ensure that the immense promised benefits of AI are available to all. 

As of late 2024, we are in the process of spinning out this work into its own non-profit entity. The Windfall Trust is building a team of interdisciplinary experts to continue to build on this vision and operationalize it into reality.

Contact: anna@futureoflife.blackfin.biz