Whether you buy AI agents or build your own, you take on the risk that they go off script. An agent might exceed its intended boundaries or follow a malicious instruction hidden in a document or CRM record. What matters is how much damage it can do when that happens.
In IBM's 2026 Cost of a Data Breach report, roughly one in five organizations reported an AI-related breach, and 92% of those lacked proper AI access controls.
What a 98% detector still misses
When TechCrunch asked me about those breakouts, I said the fix is to "put the agent in a box" and "watch everything that crosses the boundary…no exceptions." I learned this at Google, where I helped start the Threat Analysis Group after Operation Aurora. The one hole you leave open for convenience is the one that gets used.
To see how far detection alone gets you, I replayed Swarm Traces (the public record of that attack's 53,870 actions) through an off-the-shelf classification model that was never trained on this data. It flagged 98.3% of them and still missed 899, more than half of which looked like ordinary web pages. It raised no false alarms on a smaller set of 1,456 legitimate security-testing actions (read about the method and data).
That's a good detector and a bad boundary. Attacks that look normal get through, which is why QueryStory doesn't count on detection to keep data safe.
Two controls to put in place
1. Give the agent no way out
Code an AI writes should run isolated from the application, the user's session, and any credentials. Everything that’s unnecessary should be blocked, including the network. QueryStory's agents can't reach the internet: the query engine runs with external access switched off, so the queries they generate can only connect to the data sources you have connected.
Ask: If the agent were manipulated, where could it send your data?
2. Enforce permissions where the data lives
A prompt telling the model what a user may see is a suggestion. Permissions belong in the query engine or the data platform, where they apply to every query, including one the model was tricked into writing. In QueryStory, agents can only take the actions you've explicitly allowed. But limiting the changes an agent can make still doesn't limit what it can disclose, so permissions on what the agent reads matter just as much.
Ask: Can you show me a manipulated query failing to return data the user shouldn't see?
Questions to ask any AI vendor
- If the model is manipulated, what stops it? Is it the model or the system around it?
- What information do your model providers keep? Do they train on it?
- Can you show me what the AI did, and for whom?
Ask to see each answer demonstrated. SOC 2, ISO 27001, and ISO 42001 are table stakes: no certifications, no deal. Then ask about the agent controls above. In our case, QueryStory runs on the model provider your security team has already approved, including open-source and self-hosted models, so data handling stays under terms you already govern. Every query is logged, along with the user it ran for. The vendor owns the boundaries in its product; your team owns the access policies.
Before you buy or build your next agent
If an agent in your environment followed a planted instruction tomorrow, what could it do? If you can't answer that today, start there.
Unsure how your vendor would answer these questions? Building agents in-house? Let's talk. We'll be at Gartner IT Symposium/Xpo in Orlando, October 19-22. Visit us at booth #525, or reach out to our team directly at [email protected].