From Software to Silicon: Scaling Agentic AI Security

Frontier AI labs, including OpenAI and Anthropic, have recently started reporting incidents where agents have managed to slip out of their confined testing environments and perform out-of-bounds actions. One example is when Anthropic was evaluating Claude Mythos Preview, where the model was instructed to break out of its secured, or at least previously assumed secure, sandbox. The model succeeded, building a multi-step exploit to gain internet access and then emailing the researcher running the test. The most concerning part came after that, when the model, without being instructed to, posted details of its exploit on several very hard-to-find public websites.

OpenAI experienced a similar problem with Hugging Face, where a GPT-5.6 Sol and an unreleased research model have been running an internal cybersecurity evaluation with fewer guardrails than typical models have. The model found vulnerabilities in some software packages, used that to gain internet access, and stacked multiple vulnerabilities it found in Hugging Face production infrastructure to get answers to ExploitGym–the benchmark that this evaluation was supposed to solve.

While this is not a post about AI doomerism, it is an informative piece that is dedicated to showing how agentic AI security actually needs a proper and dedicated infrastructure beyond what we traditionally knew with the old model of system security. Specifically, we explore how AI agents work, why they drift off course, and what infrastructure they need to have truly functional security so they can be deployed without issues, using NVIDIA's recently published OpenShell agent safety platform as a reference design.

Agentic AI is vastly different compared to what standard chatbot interfaces are and what the majority of the public is used to. When working with humans, AI agents collect data about the task's actual goal. Based on human input, agents can adjust their mechanisms to plan and deploy a solution using a wide range of available tools. First, the agent uses a goal and breaks it down into steps, entering a loop that starts with a harness. The harness, the software wrapped around the AI model, sends the model the goal along with everything gathered so far. The model decides on what the next step should be, and the harness turns that decision into a tool call. This can be a simple web search, a file read, a shell command, or an API request. Whatever comes back is appended to the agent's context, its working memory, and the cycle starts again. It repeats until the model concludes that the goal has been met.

The agent loop is actually a simple collection where an agent uses a model to think, a harness to act with tools, and returns information back to the AI model for observing and updating the context. Inside this loop, two major parts are important for agentic AI security. The first is that the agent writes its own plan at runtime, so there is no fixed sequence of actions anyone can review in advance. The second is that every iteration passes through a model call. Almost every action starts with a model decision, which, as we’ll see, makes the connection to the model one of the most valuable checkpoints in the whole system.

The problem with AI agents starts when they start slowly branching away from their tasks or constraints that were given in the first place. For example, when you give an agent a task like building a website, it might find that using tools to open and test out the full stack might be limited, and it starts requiring permissions. Once these permissions are given, the AI agent can execute tool calls on its own. However, when the tool is missing, the agent might start looking for ways to bypass that, install a tool on its own, or even build a required tool. These small drifts away from the original goal compound, and many compounded drifts can cause an agent to go completely off course.

Policy denials are also a big issue, as agents running in loops can try to bypass policy. Not because it inherently wants harm, but because it wants to get the job done and achieve its “/goal” set by the harness. Especially when the agent is left running for hours, context is constantly getting compressed, and the original goal might not move into the context window every time an agent uses a model.

Model safety training can reduce how often agents drift, and frontier labs have made great progress there as models got smarter. What training can’t do is rule drift out completely, at least not without also dulling the capability that makes agents worth deploying, since the same creativity that solves hard problems is what finds the gaps we’re trying to close. That doesn’t make agents uncontrollable, but it does mean control can’t rely on the model alone.

Inherently, agents try to brute-force their solution by creating loops of steps that try to find and fix the issue regardless. NVIDIA’s OpenShell solution creates dedicated sandboxed environments with kernel-level isolation, which will give agents just enough freedom to execute, but not compromise the entire system. But the actual security needs to happen in every layer of the stack, propped up by each layer carrying its own walls.

The model’s job is to act as a brain and control behavior, deciding whether any action should happen. Frontier labs don't leave these models unchecked, as they spend massive resources on model safety training, implementing guardrails, and testing for malicious capabilities. Through prompt refusals, the model can decide that an action is commonly known to be for bad intent, and the model simply refuses to execute the task. Most of the frontier models are post-trained with some general guardrails that make them refuse malicious attacks. When a model refuses, that line of attack stops, but only at that step. Prior work remains, the harness can keep going, and if the model's context has been manipulated, it may not refuse at all.