Building Production-Safe AI Support Agents: Architecture, Safety Guardrails & Real-World Implementation
The engineering behind an autonomous agent that investigates AWS incidents and the guardrails that make it safe to put in front of production.
Cloud support has always worked, based on your description. Here's what changes when it can see your environment instead.
It's 2:14 AM on a public holiday. Your checkout service starts throwing 5xx errors, orders stop completing, and revenue flatlines. Your on-call engineer is awake, adrenaline up, staring at dashboards that all look fine. So you do the one thing you pay a managed cloud partner to make unnecessary: you open a support ticket and wait.
What happens in the next five minutes is the only thing that matters.
Incidents don't keep office hours. They surface at 2 AM on a public holiday, when your team is thinnest, and revenue loss is instant. In those moments, "we'll get back to you during business hours" isn't support. It's an apology in advance.
Cloud support has always had to work around two structural limits, and neither is about how good the engineer is. The first is access: the engineer usually can't see into your environment. They aren't inside your account, so they can't inspect what's actually running.
The second is context: no engineer can carry the full architecture and business logic of every customer's platform in their head. Your systems are built a particular way, for particular reasons, and an engineer meeting your case doesn't start with that map. So they work from what you can tell them, the symptoms, the error messages, the answers to their follow-up questions, and they cross-reference all of it against the documentation and runbooks to build a working theory.
That's why, from your side, the experience so often takes the same shape: Have you checked your security groups? Can you try restarting the instance? Here are the docs for that, let us know what you see. Working from the outside, without a live view and without your platform's blueprint, the engineer has to rebuild that context through you. Each theory gets relayed, run, and reported back, and the full picture comes together one answer at a time.
There's a reason no one can ever quote you an "average time to resolve a cloud incident": there isn't one. Every environment is different, and when the only source of truth is an external description, the work takes as long as it takes to turn that description into something conclusive. The hardest part, gathering the evidence, ends up being shared with you, the customer, for the simple reason that you are the one who can see the screen.
None of this is a knock on engineers. The best ones are remarkable at exactly this: drawing accurate conclusions from partial information and pointing you to the right next check. The bottleneck was never their ability. It was the starting position, on the outside of your environment and without your platform's full picture in hand. Change that starting position, and everything downstream changes with it.
We rebuilt that first move by changing the starting position. The agent begins every case inside your environment with read-only access to your actual account.
When a production case reaches us, the agent doesn't have to work from your description of the problem alone. It inspects the live state of your infrastructure directly, maps how your systems are actually wired right now, cross-checks what it finds against official AWS documentation, and reconstructs what actually happened, not what might have happened. It doesn't lean on a remembered picture of your platform either. It reads the real one, live, for this exact incident.
The agent comes back with a complete picture:
Every finding is tied to real evidence. No speculation, no "possible secondary causes," no homework handed back to you. One confirmed root cause beats three educated guesses.
And it's fast. The investigation itself takes about a minute. It's also strictly read-only with no access to your data: it can see how your infrastructure is configured, but never the contents of your databases, files, or secrets. It diagnoses; it never touches.
This is what changes when support becomes outcome-driven. You stop being handed a checklist and start being handed an answer.
Done right, automation like this doesn't replace expert engineers. It frees them for the work that actually needs them. The agent takes the first, time-critical pass at gathering evidence, so our specialists spend their time where human judgment genuinely matters: architecture, optimization, and the gnarly edge cases that deserve a real expert. And it means every customer gets the same rigorous, evidence-backed first response at 2 AM on a holiday as they would at 2 PM on a Tuesday.
It also changes what you should demand from a partner. We commit to a 5-minute SLA on production incidents: not five minutes to acknowledge your ticket, but five minutes to a real, evidence-backed diagnosis.
So here's the question worth sitting with: if the best your current partner can do in a crisis is ask what you've tried and point you to the documentation, it is because they are still working under the old constraint, investigating from the outside with no real view into your environment. That does not have to be your experience anymore.
The crisis at 2 AM will happen. What comes back in the next five minutes is a choice.
The engineering behind an autonomous agent that investigates AWS incidents and the guardrails that make it safe to put in front of production.
.png)
