Research summary
Research summary
Topic summary
The strongest public case in favor of There should be strong guardrails for Frontier Large Language Models.
Key research questions
- Investigate "There should be strong guardrails for Frontier Large Language Models" from this distinct point of divergence: Strongest arguments for.
Important claims and supporting citations
In July 2023, Anthropic CEO Dario Amodei testified that frontier AI models could assist in the deployment of chemical and biological weapons within two to three years.
SB 1047 proposed mandatory pre-deployment testing, third-party audits, kill switch capabilities, and whistleblower protections for models trained using over 10^26 FLOPs or costing over $100 million.
The Brookings Institution argues that limited liability shields shareholders while externalizing the cost of catastrophic model failures onto the public.
President Joe Biden signed Executive Order 14110 on October 30, 2023, invoking the Defense Production Act regarding AI safety.
The Brookings Institution argues that voluntary corporate self-regulation is insufficient because firms can weaken commitments under market pressure.
California Senate Bill 53 (Transparency in Frontier AI Act) was introduced to establish standardized transparency and safety reporting for frontier developers exceeding 10^26 FLOPs.
California Senate Bill 1047 was introduced in February 2024 by State Senator Scott Wiener.
SB 1047 passed the California Assembly (48–16) and Senate (30–9) in August 2024.
View all claims (text)
Claims
Claim text is for reading and citation. Bulk copying and AI training need a licensed maps data API key. Markdown
Frontier large language models are typically defined as systems operating at or beyond 10^25 to 10^26 integer…
Frontier large language models are typically defined as systems operating at or beyond 10^25 to 10^26 integer or floating-point operations (FLOPs).
Labels: Inferred, Low
The U.S.
The U.S. National Institute of Standards and Technology (NIST) established the U.S. AI Safety Institute to formulate technical guidelines for frontier foundation models.
Labels: Inferred, Low
In July 2023, Anthropic CEO Dario Amodei testified that frontier AI models could assist in the deployment of…
In July 2023, Anthropic CEO Dario Amodei testified that frontier AI models could assist in the deployment of chemical and biological weapons within two to three years.
Labels: Source-linked, High
The UK Department for Science, Innovation and Technology published an international AI safety report in 2025.
The UK Department for Science, Innovation and Technology published an international AI safety report in 2025.
Labels: Source-linked, Low
The Frontier Model Forum was formed in July 2023 by leading AI labs to advance standardized capability evalua…
The Frontier Model Forum was formed in July 2023 by leading AI labs to advance standardized capability evaluations and safety benchmarks.
Labels: Source-linked, Low
At the May 2024 Seoul AI Summit, 16 AI developer organizations signed the Frontier AI Safety Commitments, ple…
At the May 2024 Seoul AI Summit, 16 AI developer organizations signed the Frontier AI Safety Commitments, pledging to publish safety frameworks and halt deployment if risks breach thresholds.
Labels: Inferred, Medium
The Bletchley Declaration, signed by 28 nations in November 2023, established the necessity of international…
The Bletchley Declaration, signed by 28 nations in November 2023, established the necessity of international cooperation on frontier AI safety.
Labels: Inferred, Low
The Senate Judiciary Committee held hearings regarding the oversight of AI principles for regulation.
The Senate Judiciary Committee held hearings regarding the oversight of AI principles for regulation.
Labels: Source-linked, Low
Strong guardrails are intended to protect critical infrastructure, such as power grids and financial systems,…
Strong guardrails are intended to protect critical infrastructure, such as power grids and financial systems, from autonomous systemic disruptions.
Labels: Inferred, Medium
California Senate Bill 53 (Transparency in Frontier AI Act) was introduced to establish standardized transpar…
California Senate Bill 53 (Transparency in Frontier AI Act) was introduced to establish standardized transparency and safety reporting for frontier developers exceeding 10^26 FLOPs.
Labels: Source-linked, Medium
In May 2023, over 700 AI scientists, researchers, and tech executives signed a public statement identifying A…
In May 2023, over 700 AI scientists, researchers, and tech executives signed a public statement identifying AI extinction risk as a global priority comparable to pandemics and nuclear war.
Labels: Inferred, Medium
The EU AI Act (Regulation (EU) 2024/1689) classifies models exceeding 10^25 cumulative training FLOPs as carr…
The EU AI Act (Regulation (EU) 2024/1689) classifies models exceeding 10^25 cumulative training FLOPs as carrying systemic risk.
Labels: Inferred, Medium
Tracking compute infrastructure for high-end processors (GPUs and TPUs) is proposed as a method to enforce sa…
Tracking compute infrastructure for high-end processors (GPUs and TPUs) is proposed as a method to enforce safety rules without intrusive surveillance.
Labels: Inferred, Medium
The Center for AI Safety issued a statement on AI risk advocating for guardrails.
The Center for AI Safety issued a statement on AI risk advocating for guardrails.
Labels: Source-linked, Low
President Joe Biden signed Executive Order 14110 on October 30, 2023, invoking the Defense Production Act reg…
President Joe Biden signed Executive Order 14110 on October 30, 2023, invoking the Defense Production Act regarding AI safety.
Labels: Source-linked, Medium
AI Safety Risks and Capabilities
AI Safety Risks and Capabilities
Labels: Inferred, Low
Brookings Institution Arguments
Brookings Institution Arguments
Labels: Inferred, Low
California Senate Bill 1047
California Senate Bill 1047
Labels: Inferred, Low
Model Auditing and Evaluation
Model Auditing and Evaluation
Labels: Inferred, Low
Mandatory third-party model auditing is argued to protect enterprise deployers from inheriting liability and…
Mandatory third-party model auditing is argued to protect enterprise deployers from inheriting liability and security vulnerabilities from model providers.
Labels: Inferred, Medium
Anthropic's Responsible Scaling Policy (RSP) defines AI Safety Levels (ASL) and commits to pausing deployment…
Anthropic's Responsible Scaling Policy (RSP) defines AI Safety Levels (ASL) and commits to pausing deployment if biosecurity or cyber risk standards are not met.
Labels: Inferred, Medium
California Senate Bill 1047 was introduced in February 2024 by State Senator Scott Wiener.
California Senate Bill 1047 was introduced in February 2024 by State Senator Scott Wiener.
Labels: Source-linked, Medium
The Brookings Institution argues that voluntary corporate self-regulation is insufficient because firms can w…
The Brookings Institution argues that voluntary corporate self-regulation is insufficient because firms can weaken commitments under market pressure.
Labels: Source-linked, Medium
The International Scientific Report on the Safety of Advanced AI concluded that advanced AI systems present r…
The International Scientific Report on the Safety of Advanced AI concluded that advanced AI systems present risks in cyber offense, automated weaponization, and autonomous evasion.
Labels: Inferred, Medium
Independent pre-deployment evaluations are recommended because developers face conflicts of interest when eva…
Independent pre-deployment evaluations are recommended because developers face conflicts of interest when evaluating their own models.
Labels: Inferred, Medium
Frontier models may lower barriers for non-state actors to plan biological and cyber assaults by automating i…
Frontier models may lower barriers for non-state actors to plan biological and cyber assaults by automating information retrieval and troubleshooting.
Labels: Inferred, High
Governor Gavin Newsom vetoed SB 1047 on September 29, 2024.
Governor Gavin Newsom vetoed SB 1047 on September 29, 2024.
Labels: Inferred, High
SB 1047 was co-sponsored by the Center for AI Safety (CAIS) Action Fund, Economic Security California Action,…
SB 1047 was co-sponsored by the Center for AI Safety (CAIS) Action Fund, Economic Security California Action, and Encode Justice.
Labels: Inferred, Low
The Brookings Institution argues that limited liability shields shareholders while externalizing the cost of…
The Brookings Institution argues that limited liability shields shareholders while externalizing the cost of catastrophic model failures onto the public.
Labels: Source-linked, High
Ex-post legal remedies are argued to be insufficient for compensating irreversible systemic damage, such as t…
Ex-post legal remedies are argued to be insufficient for compensating irreversible systemic damage, such as the engineering of a novel pathogen.
Labels: Inferred, High
SB 1047 proposed mandatory pre-deployment testing, third-party audits, kill switch capabilities, and whistleb…
SB 1047 proposed mandatory pre-deployment testing, third-party audits, kill switch capabilities, and whistleblower protections for models trained using over 10^26 FLOPs or costing over $100 million.
Labels: Source-linked, High
SB 1047 passed the California Assembly (48–16) and Senate (30–9) in August 2024.
SB 1047 passed the California Assembly (48–16) and Senate (30–9) in August 2024.
Labels: Source-linked, Medium
Opposition to SB 1047 included Andreessen Horowitz (a16z), the Chamber of Progress, TechNet, and Meta.
Opposition to SB 1047 included Andreessen Horowitz (a16z), the Chamber of Progress, TechNet, and Meta.
Labels: Inferred, High
Geoffrey Hinton, Yoshua Bengio, and Stuart Russell formally endorsed the guardrails proposed in SB 1047.
Geoffrey Hinton, Yoshua Bengio, and Stuart Russell formally endorsed the guardrails proposed in SB 1047.
Labels: Inferred, Medium
Advanced LLMs possess multi-modal planning and code execution capabilities that could potentially automate ne…
Advanced LLMs possess multi-modal planning and code execution capabilities that could potentially automate network exploitation and discover zero-day vulnerabilities.
Labels: Inferred, High
Model safety cannot be verified by source code inspection alone, necessitating dynamic adversarial red-teamin…
Model safety cannot be verified by source code inspection alone, necessitating dynamic adversarial red-teaming to discover hidden hazardous capabilities.
Labels: Inferred, Medium
Red-teaming protocols include testing for "autonomous replication" capabilities to prevent models from acquir…
Red-teaming protocols include testing for "autonomous replication" capabilities to prevent models from acquiring resources without human oversight.
Labels: Inferred, High