Congress has spent much of the AI boom wrestling with problems that are already visible: deepfakes, job disruption, discrimination, privacy and cybersecurity.
Now lawmakers are being pressed to confront a harder category of risk — one that may never materialise, but which some researchers argue would be extraordinarily costly to ignore.
Current and former researchers working on advanced artificial intelligence have warned that future systems could become increasingly autonomous and difficult to control. Some have gone considerably further, assigning meaningful probabilities to catastrophic outcomes, including human extinction.
Those estimates are not established forecasts, and there is no scientific consensus that such an outcome will occur. Their influence in Washington nevertheless reflects a change in the regulatory conversation. Congress is being asked to decide how much evidence it should require before imposing safeguards on technology that is advancing quickly.
Researchers Put Numbers on an Uncertain Risk
Among those raising concerns is Jacob Coxon, a former Anthropic researcher who has also worked at OpenAI.
Coxon resigned and publicly discussed his concerns about the direction of advanced AI development, including the possibility that increasingly capable systems could eventually create catastrophic risks if humans lose effective control over them.
Anthropic researcher Evan Hubinger has offered an unusually stark assessment, putting the probability of advanced AI causing human extinction at greater than 10%.
The number needs context.
It is Hubinger's assessment of an uncertain future risk, not a measured probability accepted across the AI research community. Experts disagree substantially about whether today's development trajectory could lead to such systems, how quickly that might happen and how much weight policymakers should place on extreme scenarios.
That disagreement is central to the policy problem.
If researchers raising the alarm are wrong, aggressive regulation could impose substantial costs on a technology with significant economic and scientific potential. If they are broadly right, waiting for conclusive evidence could leave regulators responding only after dangerous capabilities have emerged.
Congress Finds Common Concern, but Not a Common Solution
The possibility of more powerful and autonomous systems has attracted attention across party lines, though bipartisan concern should not be confused with bipartisan agreement on regulation.
Democratic Senator Mark Kelly has argued for greater attention to the risks associated with advanced AI. Republican Senator Ted Cruz has also discussed legislation addressing AI risks, while Republican Representative Anna Paulina Luna has called for Congress to take the issue more seriously.
Senator Bernie Sanders is preparing legislation aimed at halting the development of so-called superintelligent AI until federal safety standards are in place.
The proposals circulating in Washington vary considerably. Ideas include mandatory safety evaluations, independent testing and stronger federal oversight. More aggressive approaches contemplate mechanisms that could allow authorities to intervene when a system presents an extreme threat.
Each raises difficult questions about where regulatory authority should begin and how it would work in practice.
AI Agents Give Lawmakers a More Immediate Problem
The debate is not confined to hypothetical superintelligence.
AI companies are increasingly building agents designed to carry out multi-step tasks, interact with software and operate with less direct human supervision. That development has given policymakers more immediate examples through which to examine autonomy and control.
Recent congressional scrutiny has included an incident involving OpenAI systems and AI platform Hugging Face.
Republican Senator Josh Hawley sought information about the episode, while Democratic Senator Chris Van Hollen called for federal cybersecurity authorities to receive access needed to assess the safety of OpenAI's technology.
OpenAI said it investigated the incident and has been strengthening its safeguards.
Nothing about the episode establishes that current AI systems have escaped human control. Its relevance is narrower: as AI agents receive greater ability to interact with external systems, mistakes or unexpected behaviour can have consequences beyond an incorrect answer in a chatbot window.
That creates a regulatory problem lawmakers can examine without making assumptions about what future superintelligence might look like.
Regulation Is Shifting From Products to Capabilities
Traditional technology rules often begin with a particular product or use.
Advanced AI complicates that model because one general-purpose system can perform a wide range of tasks. The same model may write routine business documents, generate software code or demonstrate capabilities relevant to cybersecurity and scientific research.
That has encouraged interest in regulation based on what a system can do rather than simply what category of product it belongs to.
Under a capability-based approach, additional obligations could take effect once a model crosses defined thresholds.
Developers might be required to conduct safety testing before deployment, submit certain models for independent evaluation, strengthen cybersecurity or report serious incidents to regulators.
The difficult part is defining the threshold.
A standard set too low could capture ordinary AI development and impose unnecessary compliance costs. One set too high could become relevant only after potentially dangerous capabilities are already widely available.
AI Companies Want Rules Too, but Details Matter
The debate is not simply a contest between regulators seeking restrictions and technology companies opposing them.
OpenAI has called for mandatory national AI safety rules and argued that voluntary commitments alone are insufficient as systems become more capable. Anthropic has also supported government involvement in frontier AI safety.
Industry support does not resolve the harder disagreements.
Companies, researchers and policymakers still differ over who should conduct evaluations, what test results should be disclosed, when government agencies should gain access to proprietary systems and whether regulators should ever have authority to delay the release of a model.
Those decisions carry commercial as well as safety consequences.
Frontier models require enormous investment, and US policymakers are simultaneously concerned about maintaining technological leadership against China. Rules designed to reduce risk could also influence where companies invest, how quickly models reach the market and whether American developers retain an advantage.
States Are Filling Part of the Federal Vacuum
Congress is not working from an entirely blank regulatory landscape.
California has moved ahead with requirements affecting developers of advanced AI systems, including safety-related obligations and scrutiny of how companies manage serious risks.
State action has created another argument for federal legislation.
Supporters of national standards say a federal framework could give developers consistent obligations rather than forcing them to comply with a patchwork of different state rules. Others argue that states should retain room to impose stronger protections when Congress moves slowly.
The longer Washington takes to establish a comprehensive approach, the more consequential that federal-state divide becomes.
Extinction Predictions Need Precision, Not Dismissal or Hype
The most severe AI warnings present a particular challenge for journalists, policymakers and the public.
A prediction involving human extinction can easily dominate discussion simply because of its scale. But treating a researcher's probability estimate as an established forecast would misrepresent the evidence.
There is no proof that today's AI systems are approaching an extinction-level capability, nor is there agreement among researchers that such an outcome is likely.
The policy case for examining the risk rests on a different argument.
Supporters of precaution contend that governments routinely prepare for events that are unlikely but potentially catastrophic. From that perspective, uncertainty is a reason to investigate safeguards before systems become more capable.
Critics counter that an excessive focus on speculative scenarios can divert political attention from problems AI is already producing and give regulators poor foundations for writing enforceable rules.
Congress does not necessarily have to choose between those priorities. Rules governing current harms can coexist with research and safeguards aimed at more advanced systems.
Washington Has to Decide What Justifies Intervention
The United States still lacks a comprehensive federal law governing frontier artificial intelligence.
That leaves lawmakers trying to design rules while the underlying technology continues to change.
They must decide which capabilities should trigger additional scrutiny, whether independent testing should be mandatory, what incidents companies must report and which agency should enforce those requirements. Any framework also has to contend with national-security concerns, commercial competition and the possibility that overly rigid rules could become obsolete quickly.
The latest warnings do not answer those questions.
What they change is the timing of the argument.
Congress is no longer being asked only to respond to documented harms after they occur. Some researchers want lawmakers to establish safeguards for capabilities that do not yet exist at the level they fear.
That leaves Washington with a difficult regulatory calculation: not whether an extreme AI scenario has been proven, but how much uncertainty the government is prepared to tolerate before requiring the companies building increasingly powerful systems to demonstrate that those systems can be controlled.
