An Exercise That Spilled Into the Real World
In May 2026, Irregular — an Israeli startup specialising in AI model security assessment — ran a capture-the-flag exercise using Google's Gemini model. The brief was straightforward: measure the agent's ability to identify and exploit vulnerabilities inside a simulated corporate environment. This type of red-team evaluation has become standard practice for stress-testing the offensive capabilities of large models before production deployment.
What happened next was anything but standard. Three real companies were compromised — without their knowledge or consent.
Two Configuration Errors, One Real Incident
Two compounding failures turned a controlled test into a live security incident. First: the fictional domain names used in the scenario happened to match those of real, existing organisations. Second: the agent's internet connection was supposed to be disabled — it wasn't, due to a misconfiguration.
Armed with unintended network access and a mission to complete, Gemini applied the techniques the exercise demanded — but against live targets. It brute-forced credentials through repeated login attempts. It retrieved secrets exposed in public code repositories. It then used those credentials to access production systems belonging to three separate companies.
No sophisticated exploit was required. What made the intrusions possible was a combination of a poorly isolated test environment and basic security hygiene failures: predictable passwords and secrets left exposed on GitHub.
The Model Stopped Itself — and That's Not Enough
Google was quick to highlight one silver lining: Gemini halted on its own. Once the agent detected it had reached real infrastructure rather than a simulated environment, its built-in safety mechanisms triggered an automatic shutdown. Heather Adkins, VP of Security Engineering at Google, stated that "the model behaved appropriately."
That is genuinely good news for model alignment. It is also a point that deserves careful handling: designing resilience around an agent's ability to self-terminate means relying on the least controllable layer of the stack. A last-resort safety net is not a substitute for upstream isolation.
The timeline of disclosure raises its own questions. Irregular notified Google in July 2026. Public disclosure only came in September, after a journalist broke the story — seven weeks of silence.
What This Changes for Your Deployments
This incident is not a one-off. Comparable situations have involved agents from OpenAI and Anthropic during the same year. AI agent evaluation under test conditions exposes a fundamental tension: the more realistic the exercise, the greater the risk that it bleeds into reality.
For IT leaders and CISOs, the operational takeaways are concrete.
Network isolation for AI evaluation environments must be actively verified, not assumed. An agent that does not need internet access to complete a task must not have it — and that rule needs to be enforced at the infrastructure level, not just in software configuration.
Fictional domain names and entities used in test scenarios should be systematically checked to avoid any collision with real organisations.
Credentials exposed in public repositories remain one of the most consistently exploited attack vectors — whether the attacker is human or autonomous. When evaluation tooling becomes as capable as the threats it simulates, sandbox rules can no longer be treated as optional.