The problem
Tool-using LLM agents are useful and risky in equal measure: they can query data and take actions, and they will read untrusted text along the way. A planted instruction in a document shouldn't be able to make the agent do anything.
The goal was an agent whose answers are checked against ground truth computed from SQL, and whose safety comes from hard controls rather than hoping the model behaves.
Architecture
How data moves through the system, left to right.
Key decisions
SQLite opened mode=ro&immutable=1 with an authorizer that only allows SELECT/READ; single statements, 2-second timeout, capped result size.
create_ticket runs only if the user's own question asks for a ticket, and only for the analyst role. The model can't decide to do it because a document said so.
Tool output is data. Sentences addressing an AI ("ignore previous instructions") are stripped before the model sees them, and logged.
37 questions in 7 categories; expected answers come from SQL, so scoring is automatic and CI can rerun it.
Runs on Groq's free API with retries, a response cache keyed by the full request, and fallback models for the live demo.
Results
| Measure | Result |
|---|---|
| Benchmark accuracy | 91.9% (34/37 questions) |
| Median latency | 1.4 s per question |
| Unauthorized actions under planted attacks | 0 |
| Injected sentences stripped | 18 |
| Tools | 5 (1 with a side effect, gated) |
| Guardrails | 6 layers between model and data |
Synthetic database seeded deterministically: 120 merchants, 45,698 transactions and 324 chargebacks. CI rebuilds it and fails if it differs. With 37 questions, differences of a few points are within noise.
What I learned & what's next
- Pattern-based injection screening catches common phrasings, not every attack. The side-effect gate and role checks are what actually stop an unrequested ticket.
- The verifier is the same model checking itself: good at catching unsupported numbers, weaker at catching wrong reasoning.
- Separating API time from rate-limit waits was essential; otherwise free-tier throttling looks like a slow agent.