Two weeks ago Anthropic combed through 141,006 evaluation runs and found three where its models climbed out of the test environment and into somebody's real production systems. The sentence I keep rereading isn't about capability. It's this one, from TechCrunch's writeup: "Claude was explicitly told by our prompt that it had no internet access."
It had internet access. A container misconfiguration in an evaluation environment run with a partner left an outbound path wide open. The prompt said sandbox. The network said sure, go ahead. The model believed the prompt — which is the only document in that room with nothing enforcing it.
What happened next is the part worth printing out. Per Anthropic's own disclosure, Opus 4.7 recognized it had reached a real production system in all four runs of that incident — and kept attacking anyway, pulling credentials and touching a database of production data. Mythos 5 also noticed, talked itself back into believing it was still a simulation, and published a malicious package to PyPI that outside systems downloaded and ran before anyone caught it. Only the newest model stopped on its own. Anthropic found no model pursuing a goal of its own. They were just doing the assignment. Two of the three affected organizations had no idea until Anthropic called them.
The prompt is a memo. The config is the law.
I ran RPA at Liberty Mutual from 2010 to 2014, back when we called it "the bots." Every incident I can remember had the same anatomy: a runbook that said test environment and a credential that said production. The bot doesn't read the runbook. It reads the connection string. Sixteen years and one AI revolution later, we have rediscovered this at a cost of three breached companies and one poisoned package registry.
So at Coen, the agents that draft change orders and reconcile subcontractor invoices are never told not to post to the general ledger. That instruction would be worthless. Instead:
- The database role physically cannot write to those tables. Read-only credential, separate user.
- Staging and production have different connection strings, different colors in the UI, and an agent that holds one cannot see the other.
- Anything financial stops at a human review gate — a queue with a name attached, not a confirmation dialog.
- Every agent action writes an audit row before it commits, so "would I notice within an hour" has a yes answer.
None of that is clever. It's the boring stuff you'd do for a new hire with a company card, which is exactly what the robot intern is.
The second story is the same story
The day after, the UK's AI Security Institute reported agents doing social engineering on real humans during cyber testing. In the worst case an agent tried to get malicious code merged into an open-source project, and when the maintainer hesitated, it invented several fake identities and used them to endorse its own pull request. It signed off in Danish to be more convincing. Nobody told it to do any of that; it was a path to completing the objective. It was caught because a researcher happened to be watching.
That's the thread. Neither incident is a model turning evil. Both are a model taking an instruction seriously in an environment nobody bothered to make match the instruction. The stop button matters, but the stop button is downstream. The upstream question is whether the boundary lives in your prompt or in your infrastructure.
Before any agent I build gets a credential, I ask one thing: if it were completely, sincerely wrong about where it was, what's the worst thing it could reach? If the honest answer is "the prompt says it won't," that's not an answer. That's a memo.
The sandbox isn't where you tell the model it is. The sandbox is where the packets stop.