Platform Shifts

AI safety tests fail to stop secret leaks

 ·  By Isadora Dunmore
AI safety tests fail to stop secret leaks - ai safety
AI safety tests fail to stop secret leaks

An AI agent built to automate code reviews can pass every safety check yet still leak secrets, according to new research by Novee Security founding engineer Elad Meged.

Meged evaluated default configurations of three widely used AI coding assistants: Claude Code Action, Google’s Gemini, and Codex CLI. Each agent approved commands that appeared secure but later exposed sensitive data through unexpected handoffs between approval and execution.

How the attacks worked

The agents Meged examined consist of a model paired with a harness. The model generates intent, such as “run this test,” while the harness converts that intent into shell commands, file reads, or API calls. It also enforces security rules, including permitted paths, allowed domains, and whether outputs should be published.

Prompt injection served only as the delivery method. The actual vulnerabilities emerged from how the harness made trust decisions and how those decisions compounded across stages.

“A command gets approved because it looks safe,” Meged explained. “The output gets published because that’s the default. Neither decision is wrong on its own, but together they form an exfiltration chain.”

Related: Moonshot Unveils Kimi K3 Weights to Select Few

In one test, the Claude Code Action pipeline approved a read-only command, then automatically posted its output to a public thread. The command didn’t violate any rules, but the combination of approval and publication created a leak. The same issue appeared in Google’s Gemini CLI and Codex CLI, where data written by one stage was later treated as trusted context by another.

Patches shifted the problem without solving it

After Meged disclosed the findings, vendors released fixes. Anthropic, for example, added more checks to its pipeline. Each patch, however, redefined what counted as safe, revealing new attack surfaces in the process.

“By the final round, we recovered secrets through a channel that survived every prior fix,” Meged said. “No outbound connection to an attacker, no writes, no logs—just a handoff between two stages that both assumed they were operating safely.”

Anthropic issued bounties for each finding. Meged acknowledged the practice as reasonable but noted it highlights a structural flaw. “A bounty rewards a specific bug, not the architectural seam that keeps breaking,” he said. “When the same researcher keeps finding new angles, the response is ‘we consider this closed’—even when the underlying exposure remains.”

Google took a different approach, describing its fix as an “Update to the Trust Model” in an advisory with a 10.0 CVSS severity rating. The advisory explained that a restriction configured by operators went unenforced during execution, a flaw Meged demonstrated in Gemini’s default CI workflow setup.

Related: Microsoft invests $2.5 billion in new AI division

The handoff problem

Meged identified the core issue as a tendency for agent harnesses to make trust decisions early and propagate them without revalidation. A command labeled “read-only” might be safe alone, but if its output is later published or used as configuration, the label no longer applies.

All three vendors had safeguards in place. Anthropic used layered checks, Google implemented execution modes with environment sanitization, and Codex CLI shipped a default sandbox with protected paths. Yet in each case, the defenses failed at the handoffs between stages.

For teams using these agents in production, Meged advises tracing every path where the agent’s output—or any state it influences—gets consumed by a later stage with different privileges. “Find where your harness says ‘this is safe,’” he said, “then ask what happens to that output next.”

The findings reveal a shared assumption across the industry. While individual companies can improve their systems, Meged believes the discussion must expand to include the entire sector.

He plans to present the full code-level analysis and live demonstrations at Black Hat USA 2026.

Leave a Comment

Your email address will not be published.