AI agents look magical in demos. In real operations, they’re more like a new employee who can click buttons across your systems — fast, confidently, and sometimes incorrectly. AgentOps is what makes agents safe to run in production: clear boundaries, predictable behavior, and the ability to explain what happened after the fact.
The 3 levels of automation (don’t skip a level)
Most agent incidents happen when teams jump from “cool prototype” to “full autopilot.” Use this ladder:- Level 1 — Suggest: The agent drafts and recommends. A human clicks “Send.”
- Level 2 — Assist: The agent prepares actions (tickets, updates, refunds) but still needs approval.
- Level 3 — Act: The agent executes actions automatically — within strict rules.
The AgentOps checklist: 9 questions anyone can ask
Don’t treat this as a “security list.” Treat it as operational common sense: the questions that stop expensive surprises. Tip: Read each line and answer it out loud. If you can’t answer in one sentence, you’ve found the next thing to fix.[ ] 1) What systems can the agent touch?
Be specific: email, CRM, Jira, ServiceNow, payments, HR, production, etc. If you can’t list them, you don’t control them.[ ] 2) What is it explicitly NOT allowed to do?
Write a short deny-list (e.g., “no refunds above €X,” “no user deletion,” “no changing billing details,” “no contacting customers directly”).[ ] 3) Does it run with least privilege?
If the agent has admin access “for convenience,” you’re building a future incident.[ ] 4) What does success look like (in one sentence)?
Example: “Creates correct support tickets with the right category and priority.” Without a crisp definition, quality debates never end.[ ] 5) What’s the failure plan?
When the agent is uncertain, what happens? Ask a clarifying question, escalate to a human, or stop. A quick horror story: Imagine a support agent misreads sarcasm as a serious complaint and issues a €500 credit—even though policy forbids it. Without a failure plan (limits + approvals + escalation), you might only discover it at month-end when finance asks why credits spiked.[ ] 6) What’s the rollback?
If the agent makes 200 changes, can you undo them quickly? If rollback is painful, teams avoid shipping — or ship and pray.[ ] 7) Do we have an “Agent Receipt” for every run?
A receipt is a simple record of what the agent decided and what it changed. (More on this below — it’s the most important concept.)[ ] 8) Can we answer “Why did it do that?” in 60 seconds?
If the only way to explain actions is “ask an engineer to reconstruct it,” you don’t have AgentOps — you have a demo.[ ] 9) Who owns the agent in production?
Name a real owner. Define who handles incidents, who approves changes, and who can pause automation when something looks off.
The hook: The “Agent Receipt” (your best protection against chaos)
If you implement only one AgentOps idea, implement this: Every agent action must leave a receipt. Not a vague log. A readable, auditable “what happened” record. The real job of AgentOps tooling is producing this receipt automatically so you can debug, audit, and improve without guessing.The receipt doesn’t need to be fancy. It needs to be consistent:
- ORDER: what the user asked for
- THOUGHTS (short): the rule or reason used (e.g., “eligible per policy 4b”)
- EVIDENCE: which documents/records were used (if any)
- ACTION: what systems were touched + what changed
- RESULT: success/failure + why
A safe rollout plan (no bravado required)
Want to launch without betting your reputation? Use a staged rollout:- Start at Level 1 (Suggest) for 1–2 workflows and review receipts.
- Move to Level 2 (Assist) when receipts look clean and repeatable.
- Only then consider Level 3 (Act), and keep strict limits for high-impact actions.