Points of Focus

• Researchers say AI agents should be treated as untrusted systems, not trusted decision-makers.
• Analysis of 11 real-world attacks found many could have been prevented with established security controls.
• The warning comes as companies expand AI agents into payments, coding, and enterprise workflows.

AI agents are gaining access to wallets, coding tools, business software, and financial systems, prompting researchers to question whether current security approaches are keeping pace with those capabilities.

Researchers from Google, UC San Diego, the University of Wisconsin-Madison, Gray Swan AI, EmbraceTheRed, and Cornell University have said AI agent security should be treated as a systems problem. Their research, released on May 20, found that model-level defenses alone are unlikely to address many of the security risks facing autonomous agents. The study recommends treating AI models as untrusted components and enforcing safeguards at the system level.

Researchers challenge model-first AI security approaches

The authors argue that improving model robustness remains useful but is unlikely to provide dependable protection against determined attackers on its own. Instead, they draw on decades of computer-security research that assumes software components can fail or behave unpredictably under hostile conditions.

Their analysis examined 11 real-world attacks against AI-powered systems, including incidents involving prompt injection, data exfiltration, account compromise, and unauthorized access to sensitive information. According to the researchers, many of those attacks exposed weaknesses not only in the models but also in the broader systems surrounding them.

Three security controls could reduce agent attacks

The paper identifies three mechanisms that could mitigate many of the security failures affecting AI agents today.

The first is separating instructions from untrusted data so that malicious content can’t be interpreted as commands. The second is least-privilege sandboxing, which limits an agent’s access to only the resources required for a specific task. The third is information-flow control, a technique designed to restrict how sensitive data moves through a system.

Researchers compared the approach to traditional operating-system security, where applications run inside restricted environments rather than receiving unrestricted access to a device.

AI agent adoption raises stakes for security failures

The debate comes as companies invest heavily in AI agents capable of handling payments, software tasks, and business workflows with limited human oversight. Goldman Sachs recently estimated AI-agent activity could increase sharply by the end of the decade, while crypto and technology firms are investing in infrastructure that allows agents to make payments, execute workflows, and interact with online services autonomously.

A new report from @GoldmanSachs estimates AI agent activity will increase by 2,300% by 2030.

AI agents need fast, cheap, reliable settlement to operate at scale.

Algorand has it all: instant finality, high throughput, predictable low fees, zero downtime since launch. pic.twitter.com/grLRkf7jAI

— Algorand (@Algorand) May 25, 2026

The researchers stop short of claiming system-level controls are a complete solution. The paper identifies unresolved challenges around policy enforcement, instruction verification, and tracking how information moves through AI systems. Those challenges remain unresolved even as companies expand the use of autonomous agents in payments, software development, and enterprise operations.