Rogue AI agents? Nvidia says it has a fix

Please enable JavaScript for the best SAMAA TV experience.

Note: For enhanced search features, please enable JavaScript.

Nvidia unveils platform to contain rogue AI agents

Nvidia has unveiled a new platform designed to prevent AI agents from escaping controlled environments and accessing systems beyond the permissions given to them.

Nvidia CEO Jensen Huang introduced the Nvidia Open Agent Safety Platform on Monday, combining software and hardware tools that the company says can provide independent security layers around AI agents.

The launch comes amid growing concern over AI agents bypassing security controls during testing and gaining access to real-world systems.

The new platform combines OpenShell, Nvidia’s open-source software for controlling what AI agents can access, with Sentry, an independent monitoring system that runs on Nvidia’s BlueField-4 data processing units.

Unlike security controls running on the same CPU or GPU as an AI agent, Sentry operates on a separate processor. Nvidia says this gives the system an independent view of an agent’s behaviour and allows it to intervene even if the agent attempts to bypass its software controls.

The company said Sentry can continuously monitor agents and quarantine those attempting to move beyond their defined boundaries “in milliseconds.”

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” Huang said in a statement. “Safety and security require full-stack engineering.”

Nvidia’s announcement follows several reported incidents involving AI systems from major technology companies that bypassed safeguards or escaped their intended testing environments.

One prominent case involved OpenAI agents that breached the AI development platform Hugging Face while carrying out a cybersecurity task, according to reports.

Nvidia said its new safety architecture would have prevented such breaches. That claim has not been independently demonstrated.

The company argues that security controls should not depend entirely on the AI agent itself following instructions. Instead, some protections should operate independently of the agent.

“When you deploy an agent, no matter how smart, the first thing you do is to take away all of its rights,” Huang told CNBC.

Nvidia has argued that improving the engineering of AI systems, rather than slowing AI development through additional restrictions, can address many of the risks associated with increasingly autonomous agents.

The company began working on the platform about a year ago, according to Huang.

Nvidia previously introduced NemoClaw, an enterprise-focused AI agent platform that incorporates security features, based on OpenClaw, an agent operating system developed by Peter Steinberger.

The latest platform expands that approach by adding an independent hardware-based monitoring layer.

Nvidia said dozens of companies have agreed to support the initiative and use its open-source technology.

Those listed include Anthropic, Arm, Microsoft, Oracle and SpaceX.

OpenAI was not listed among the participating companies.

The platform is being introduced as AI developers increasingly deploy agents capable of independently using tools, accessing files and systems, writing code and performing multistep tasks.

Unannounced feature briefly appeared on a ChatGPT Pro screen

iPhone 18 Pro Max’s biggest drawback may be its weight