Risk thresholds for frontier AI

Overview

• Frontier artificial intelligence (AI) systems could pose increasing risks to public safety and security
• Two approaches to managing these risks are discussed:
  1. Defining capability thresholds - describing AI capabilities beyond which a system is deemed too risky
  2. Defining risk thresholds - stating how much risk would be too much (e.g. likelihood of economic damage exceeding a certain level)
• Risk thresholds are more principled but harder to evaluate reliably
• Recommended approach is to use risk thresholds to help set capability thresholds, while primarily relying on capability thresholds for decision-making
• Regulators should explore defining risk thresholds as the most legitimate actors to do so

Plain English Explanation

As frontier AI systems become more advanced, they could start to pose serious risks to public safety and security. To manage these risks, there are a few different approaches that can be taken.

One approach is to define capability thresholds - these describe the specific capabilities that an AI system would need to have before it's considered too risky to deploy. For example, you might say that an AI system is too risky if it's capable of causing more than $X million in economic damage.

Another approach is to define risk thresholds - these directly state how much risk would be too much, rather than focusing on specific capabilities. For instance, you might say that the likelihood of an AI system being used by cybercriminals to cause significant economic damage must not increase by more than Y%.

The advantage of risk thresholds is that they're more based on principles and values, rather than just technical capabilities. However, they're also harder to evaluate accurately.

As a result, the recommended approach is for companies to:

1. Define risk thresholds to provide a principled foundation
2. Use those risk thresholds to help set appropriate capability thresholds
3. Then primarily rely on the capability thresholds when making decisions about deploying AI systems.

Regulators should also get involved in defining risk thresholds, since they're in the best position to do so in a way that's legitimate and represents the public interest. And if AI risk assessment methods become more reliable over time, risk thresholds could play an even more direct role in decision-making.

Technical Explanation

The provided paper discusses two main approaches to managing the increasing risks posed by frontier artificial intelligence (AI) systems:

1. Capability Thresholds: Defining specific AI capabilities beyond which a system is deemed to pose too much risk. For example, stating that an AI system must not be capable of causing more than $X million in economic damage.

2. Risk Thresholds: Directly defining the maximum acceptable level of risk, such as stating that the likelihood of an AI system being used by cybercriminals to cause significant economic damage must not increase by more than Y%.

The key advantage of risk thresholds is that they provide a more principled foundation based on values and priorities, rather than just technical capabilities. However, reliably evaluating and quantifying AI risks is often challenging, making risk thresholds harder to implement in practice.

As a result, the authors recommend that companies take a hybrid approach:

1. Define risk thresholds to provide a principled starting point
2. Use those risk thresholds to help set appropriate capability thresholds
3. Then primarily rely on the capability thresholds when making decisions about deploying AI systems

The paper also suggests that regulators should explore defining risk thresholds, as they are the most legitimate actors to do so in a way that represents the public interest. And if AI risk assessment methods become more reliable over time, risk thresholds could play a more direct role in decision-making.

Critical Analysis

The paper presents a thoughtful discussion of the tradeoffs between capability thresholds and risk thresholds for managing the risks of frontier AI systems. The authors rightly acknowledge the challenges in reliably quantifying AI risks, which is a significant limitation of the risk threshold approach.

One potential criticism is that the paper does not delve deeper into the specific methods and metrics that could be used to define and evaluate risk thresholds. Further research may be needed to develop more robust and standardized approaches in this area.

Additionally, the paper does not address the complexities of how risk thresholds could be implemented in practice, such as the potential for disagreement between different stakeholders (e.g. companies, regulators, the public) on what constitutes "acceptable" risk. [Exploring the trade-offs and balancing risk and benefits could be an important area for future work.

Overall, the paper provides a useful framework for thinking about how to approach the management of AI risks, but there is still significant room for further research and practical application of these concepts, particularly as the capabilities of frontier AI systems continue to evolve.

Conclusion

The paper discusses two main approaches to managing the risks of frontier AI systems: capability thresholds and risk thresholds. While risk thresholds are more principled, they are also more challenging to evaluate reliably.

The recommended approach is for companies to define risk thresholds to provide a foundation, use those to help set capability thresholds, and then primarily rely on the capability thresholds for decision-making. Regulators should also explore defining risk thresholds, as they are the most legitimate actors to do so.

As AI risk assessment methods become more advanced, risk thresholds could play an increasingly direct role in managing the deployment of frontier AI systems. However, further research is still needed to develop more robust and standardized approaches in this area.