Building Production-Safe AI Support Agents: Architecture, Safety Guardrails & Real-World Implementation
The engineering behind an autonomous agent that investigates AWS incidents and the guardrails that make it safe to put in front of production.
The engineering behind an autonomous agent that investigates AWS incidents and the guardrails that make it safe to put in front of production.
Every managed cloud provider hits the same wall eventually. Incidents arrive around the clock, each one unique, and the quality of the response depends on getting an expert with the right context in front of the problem fast. Experts are finite. Incidents are not. And no engineer, however good, starts a case with a live view of the customer's environment or the full architecture of their platform committed to memory.
We decided to treat that as an engineering problem. The result is an autonomous support agent that investigates AWS incidents on its own: it reads the customer's environment, grounds itself in official AWS documentation, gathers evidence, and writes back a structured diagnosis, typically in about a minute. This is the story of what we built, and, just as importantly, how we made it safe enough to trust.
The goal was never "add AI to support." It was to give every customer the same fast, rigorous first response at 3 AM on a holiday that they would get from our best engineer on a quiet afternoon, and to do it without asking them to compromise on security.
That framing set four hard requirements:
It had to be autonomous for the first pass. Our team is on call around the clock, and the agent works alongside them. It picks up every case the instant it arrives, at any hour, and starts investigating immediately, so a thorough first analysis is ready within minutes.
It had to be trustworthy with access. Customers will not, and should not, hand a vendor broad permissions into production. Whatever we built had to be strictly read-only and provably unable to touch their data.
It had to be accurate and grounded. A confident, wrong answer about production infrastructure is worse than no answer at all. The system could not be allowed to guess.
It had to be safe by construction, not by hope. "We told the model to be careful" is not a safety strategy.
The hard part was never the AI. It was meeting all four of those constraints simultaneously.
At the center is an agent built on Amazon Bedrock, using Anthropic's Claude as the reasoning engine. But the model is the smallest part of the story. What makes it useful is the discipline we built around it.
The agent follows a documentation-first protocol. Before it is allowed to make a single live call against a customer's environment, it has to consult the relevant official AWS documentation. This isn't a polite suggestion buried in a prompt; it's enforced in code. If the agent tries to inspect infrastructure before grounding itself in the docs, the attempt is intercepted and redirected. Every investigation, therefore, starts from authoritative AWS knowledge rather than from the model's assumptions.
From there, it gathers evidence. It makes read-only calls to inspect the actual state of the environment, the network configuration, the resource settings, the recent change history, and builds its conclusion from what those calls return. We hold it to a strict evidence hierarchy: what the APIs return outranks everything, the absence of a problem in the data is itself a finding, and the customer's description is treated as a starting direction, never as a root cause.
Speculation is prohibited. Every finding has to point to a real result. One confirmed root cause is worth more than three plausible theories, and the agent is built to prefer the former.
This is the part that separates a system from a demo.
We enforce that on two fronts. The agent never chooses which account it investigates: every case originates in our portal, where an authenticated user can only open one against an account they already own, so the target is a trusted identity that cannot be forged, and our code, not the agent, builds the exact cross-account role from it for each case.
And the agent never holds more than it needs once inside: that role can describe and inspect configuration across services, but deliberately cannot read data, with no access to object contents, database records, secrets, or parameter values. Either way, the boundary is fixed outside the agent, and it cannot be widened from within. The agent sees how your infrastructure is built, never what's inside it. The role is assumed only for the length of an investigation, and every access is tagged and auditable.
Part of every case is free-form text written by someone outside our control: the customer's own description of the problem. That is a classic prompt-injection vector, where instructions hidden in the input try to hijack the agent. We treat that text as data, not as commands. It is screened for known injection patterns and validated against a strict schema before the agent acts on it, and it can only ever suggest where to look.
The diagnosis itself is built from the environment, never from what the agent was told.
The behaviors that matter for safety are enforced in code, not left to the model's judgment. The clearest example: if the agent loses its connection to the documentation source and can no longer ground its reasoning, hard-coded logic caps its confidence, flags the case for human review, and marks the output accordingly. The model cannot override this. The system is designed to know when it isn't sure, and to hand off when it isn't.
Cases reach the agent over a mutually authenticated channel and are encrypted at the application layer. The agent runs in an isolated runtime, maintains no persistent state between cases, and keeps secrets entirely out of its infrastructure code. Each investigation is self-contained. It reads the real environment, reasons, reports, and gets terminated.
Every diagnosis carries a confidence score. On well-scoped incidents, the agent typically lands between 85 and 98 percent, and when it can't, the same guardrails route the case to a person. Honesty is the point. We would rather the system say "I'm not certain, look here" than dress up a guess as an answer.
The output is not a chat transcript. It's a structured diagnosis written straight back to the case, in the same shape every time: what happened, what changed, why it happened, and how to fix it, with each line tied to the specific evidence behind it.
A representative example (illustrative, not a real customer case): an instance loses outbound connectivity. The agent inspects the environment and finds that the private subnet's route table has no path to the internet and the security group has no outbound rules, cross-references the relevant AWS networking documentation, points to the change in the recent history that introduced the problem, and lays out the exact ordered steps to restore connectivity, each one backed by the calls it made to confirm it. Total time: about a minute.
The difference from the traditional model is the starting position. Instead of working from a description and a stack of documentation links, the agent works from the live environment itself and returns an answer rather than a checklist.
What we set out to build wasn't a feature. It was a complete system that investigates on its own: it takes in a real incident, examines a live production environment it has never seen before, and reaches a defensible conclusion without a person driving each step. But whether a case goes out or gets escalated to a human isn't the model's call.
That decision is made in code: defined conditions, such as confidence falling short or the agent losing its grounding in the documentation, force a handoff that the model cannot talk its way past. The autonomy is real, but it stops exactly where the system says it does, not where the model feels confident.
That is the real point. The hard part of putting AI into production isn't calling the model; it's everything that has to hold around it: scoping access so the agent can never overreach, grounding it so it can't invent facts, building guardrails the model itself cannot bypass, and tying every conclusion to the evidence that proves it.
We designed, secured, and operate all of it, and turning a real business problem into a system that does this safely in production is exactly the kind of work we do for customers.
If this is what we build for our own hardest problem, it's a reasonable baseline for what we can do for yours.
The agent's role can inspect configuration across services but cannot read data. It has no access to object contents, database records, secrets, or parameter values, and that boundary is fixed outside the agent. Every case also runs over an encrypted, mutually authenticated channel in an isolated, stateless runtime.
Four things at once: autonomous from the first pass, scoped to no more access than it needs, every conclusion grounded in evidence rather than guessed, and safety enforced in code, not just prompted. Guardrails like forced human review when the agent loses its documentation grounding are what make that autonomy safe.
By treating the customer's incident description as data rather than instructions. That text is screened for injection patterns and validated against a strict schema before the agent acts on it; it can only suggest where to look, never dictate the diagnosis.
Here, autonomous means the agent investigates and writes the full diagnosis on its own, without a person driving each step. That autonomy is bounded: defined conditions, such as low confidence or lost documentation, are hard-coded to force a handoff to a human.
The engineering behind an autonomous agent that investigates AWS incidents and the guardrails that make it safe to put in front of production.
.png)
