AI SRE agents on call: an autonomy ladder for incident response

I've carried the pager in two very different settings. At Avizva, for SOC 2-compliant US health insurers, we held a 99.8% availability SLO and kept MTTR to about 30 minutes, with Zenduty handling routing and escalation. At Deutsche Telekom Digital Labs, I was on call during live FIFA World Cup 2026 matches, on a team spread across India and Germany. In both places, the first ten minutes of an incident looked the same: open five dashboards, grep logs, check what was deployed, and try to form a hypothesis before the next person joins the bridge.

That first ten minutes is exactly what AI SRE agents promise to compress. Some of them deliver. But an agent that can query your cluster can also, if you let it, change your cluster. So before choosing a tool, I think every SRE team needs a written answer to one question: at each level of trust, what may the agent do, what must it prove, and who takes over when it's wrong?

1. Why this is a real decision now

A year ago this was a demo topic. In 2026 it's a procurement decision:

• AWS DevOps Agent went GA on 31 March 2026, positioned to triage incidents across AWS, multicloud and on-prem, using your observability tools, runbooks, code and CI/CD data (AWS announcement).
• HolmesGPT is a CNCF Sandbox project (accepted October 2025). It breaks an investigation into tasks, queries Prometheus, Kubernetes events and logs through extensible toolsets, including MCP servers, and returns a diagnosis (CNCF blog).
• Agents are becoming Kubernetes workloads. kagent runs agents as custom resources, and the CNCF tutorial's example agent uses a three-tool allowlist and human approval before applying changes (The New Stack, 16 Oct).
• The industry is starting to talk in phases of autonomy. A New Stack piece this month describes three phases: invoked (you ask it), background (rules and schedules trigger it) and proactive (it decides an incident needs attention), and notes most tools still leave the key decision to humans (The New Stack, 8 Oct).

Meanwhile the evidence on AI and delivery is mixed. The DORA 2025 report frames AI as an amplifier: it makes strong engineering systems stronger and weak ones more chaotic. That matches my instinct for operations. An agent on top of good runbooks, clean telemetry and clear ownership will help. An agent on top of a messy platform will produce confident nonsense faster.

2. What an agent is actually good at in an incident

Break an incident into the tasks humans do, and the fit becomes clearer. Agents are strong at reading lots of data fast and writing it up. They're weak at judgement under ambiguity and at knowing what they don't know.

Notice the pattern: the value is front-loaded in reading and writing, and the risk is concentrated in acting. That's why a single "autonomous: yes or no" switch is the wrong design. You want a ladder.

3. The autonomy ladder

Self-driving cars got a shared vocabulary from the SAE levels, and it made the arguments far more productive. Incident response needs the same. Here is the ladder I'd use. The key idea is that it measures authority (what the agent may change), which is a separate axis from the trigger (whether you invoke it, a rule fires it, or it acts proactively). A proactive agent can still be read-only, and that's often exactly what you want.

Two rules make the ladder work in practice. First, levels are per action type, not per agent. An agent can be at L4 for "restart one crash-looping Deployment" and at L2 for "fail over the database writer" forever. Second, demotion is automatic. One harmful action drops that action type back to L3 until a human reviews it. That's how you keep trust earned rather than assumed.

4. Where the agent sits in the incident flow

The architecture that holds up is one where the agent never holds write credentials itself. It reads through a gateway, and it asks an action broker to change things. The broker checks policy, gets approval if needed, executes with its own narrowly scoped identity, and reports back. This is the same separation I argued for with agents that write Terraform: the thing that thinks shouldn't be the thing that holds the keys.

Some details in that flow that matter more than they look:

• The human is paged at the same time as the agent starts. Never route the page through the agent ("let the AI try first for 10 minutes"). If the agent hangs, loops or gets confused, you've just added 10 minutes to your MTTR.
• The gateway is the audit point. An MCP gateway in front of tools gives you an approved catalogue of servers, per-agent access rules and a full log of every call. The Agentic AI Foundation describes this pattern, along with sandboxing the agent at the OS level, after incidents where permitted tools still did real damage.
• Rollbacks go through Git. With Argo CD, the cleanest mitigation is a git revert of the deploy commit that Argo CD then syncs. The change is reviewed, logged and reversible in the same way as any other deploy, and nobody fights auto-sync at 3 AM.
• Everything lands in the incident channel. Every query, hypothesis, proposal and action is posted where humans already look. If it's not in the timeline, it didn't happen.

5. Read-only by default: the agent's identity

L1 sounds safe because it's "read-only". But read-only is not the same as harmless. An agent that can read Secrets, exec into pods or decrypt with KMS can leak far more than it can break. The Kubernetes RBAC good practices page is explicit that list and watch on Secrets reveal their contents, and that pods/exec and nodes/proxy are effectively escalation paths.

Here's the ClusterRole I'd start with. It's deliberately boring: it reads what an investigator needs and nothing that holds data or credentials.

# ai-sre-reader: what an investigating agent can see in the cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-sre-reader
rules:
  - apiGroups: [""]
    resources: [pods, pods/log, events, services, endpoints, nodes, namespaces, persistentvolumeclaims]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, replicasets, statefulsets, daemonsets]
    verbs: [get, list, watch]
  - apiGroups: [autoscaling, policy, events.k8s.io]
    resources: [horizontalpodautoscalers, poddisruptionbudgets, events]
    verbs: [get, list, watch]
  - apiGroups: [keda.sh, karpenter.sh, argoproj.io]
    resources: [scaledobjects, nodepools, nodeclaims, applications]
    verbs: [get, list, watch]
# Deliberately absent: secrets, configmaps, pods/exec, pods/portforward,
# nodes/proxy, serviceaccounts/token, and every write verb.

On EKS, bind this to the agent's IAM role with an access entry, or start from the managed AmazonEKSViewPolicy, which also excludes Secrets (EKS access policies). ConfigMaps are left out on purpose: in real clusters they too often hold connection strings that should have been Secrets.

The cloud side follows the same idea. Allow the specific read calls an investigator needs, and add an explicit deny for anything that returns secret material, so a later "just attach ReadOnlyAccess" doesn't quietly widen it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvestigateTelemetryAndInventory",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricData", "cloudwatch:DescribeAlarms",
        "logs:StartQuery", "logs:GetQueryResults", "logs:FilterLogEvents",
        "cloudtrail:LookupEvents", "health:DescribeEvents",
        "eks:DescribeCluster", "ec2:DescribeInstances",
        "elasticloadbalancing:DescribeTargetHealth", "rds:DescribeDBInstances"
      ],
      "Resource": "*"
    },
    {
      "Sid": "NeverReadSecretMaterial",
      "Effect": "Deny",
      "Action": [
        "secretsmanager:GetSecretValue", "ssm:GetParameter*",
        "kms:Decrypt", "s3:GetObject", "rds-db:connect"
      ],
      "Resource": "*"
    }
  ]
}

Two more controls belong here. Use short-lived credentials (EKS Pod Identity or OIDC federation), never a long-lived key in the agent's config. And redact before the model sees it: logs contain tokens, emails and customer IDs, and for regulated clients like the health insurers I worked with, that alone decides whether a hosted model is allowed at all.

6. Acting safely: an action catalogue and a policy gate

Above L2, the agent stops writing free-form commands. It can only pick from a catalogue of actions that humans wrote, reviewed and tested, each with its preconditions, blast radius, verification and rollback. This is the same idea as Google's advice to automate the response to known failure modes rather than improvise (SRE book, Eliminating Toil). The agent's job becomes choosing the right runbook step, not inventing one.

# runbook-actions.yaml: the only things the broker will ever execute
actions:
  - id: restart-deployment
    does: "kubectl rollout restart deployment/{name} -n {namespace}"
    reversible: true
    blast_radius: single-workload
    max_level: 4
    namespaces: [checkout, catalogue, notifications]
    preconditions: [pdb_allows_disruption, ready_replicas_ratio_gte_0_5]
    verify: { window: 10m, slo: "5xx ratio and p99 back under target" }

  - id: revert-last-deploy
    does: "git revert the last deploy commit; Argo CD syncs it"
    reversible: true
    blast_radius: single-service
    max_level: 3                     # always needs a human click
    preconditions: [deploy_within_last_2h, no_schema_migration_in_release]

  - id: fail-over-database
    does: "promote replica, repoint writers"
    reversible: false                 # fail-back is a separate, planned job
    blast_radius: many-services
    max_level: 2                     # recommend only, forever

The broker then asks a policy engine whether this specific request is allowed right now. I like OPA here because the rules are code, testable in CI, and the same tool many teams already use for admission control. The policy below returns one of three answers: auto, needs_approval or deny.

package incident.actions

# Input: { action: {id}, agent: {level}, target: {namespace},
#          freeze: bool, evidence: [...], recent_actions: [...] }
# data.catalogue: runbook-actions.yaml, keyed by action id

default decision := "deny"

action := data.catalogue[input.action.id]

decision := "auto" if {
	auto_ok
} else := "needs_approval" if {
	approval_ok
}

auto_ok if {
	approval_ok
	action.reversible
	action.blast_radius == "single-workload"
	action.max_level >= 4
	input.agent.level >= 4
	not input.freeze
	recent_auto_actions < 2               # stop loops: max 2 auto actions per incident
	count(input.evidence) >= 2            # two independent signals, not one
}

approval_ok if {
	action.max_level >= 3
	input.agent.level >= 3
	input.target.namespace in action.namespaces
	count(input.evidence) >= 1
}

recent_auto_actions := count([a | some a in input.recent_actions; a.mode == "auto"])

That rate limit on auto actions is the most important line in the file. Automated remediation that keeps retrying is how a small incident becomes a large one. The AWS us-east-1 DynamoDB event of October 2025 is a good public reminder: there was no AI involved, but two automated DNS processes racing each other removed every IP address for the regional endpoint, and AWS's response was to disable that automation worldwide until it had a fix and new safeguards. Automation needs brakes, and an AI agent is automation that can also be confidently wrong.

Here's the decision tree the policy encodes, which is also how I'd explain it to an incident commander:

7. How AI SRE agents fail

The vendors' own data is honest about the biggest limitation. Traversal's 2026 review of the field argues that most AI in incident response speeds up correlation, not causal root-cause analysis: "correlation tells you what broke together, causation tells you why" (Traversal). Here are the failure modes I'd design against, and the control for each.

The last one is the oldest lesson in operations. Lisanne Bainbridge's "Ironies of Automation" (1983) pointed out that the more you automate, the more you depend on humans to handle the rare cases automation can't, while giving them less practice. An AI SRE agent makes that irony sharper, not weaker.

8. Measuring it, and when to promote a level

"MTTR went down" is too blunt to prove an agent helped, because MTTR moves with incident mix. Split the timeline and measure the agent's contribution directly. The SRE Workbook chapter on incident response and NIST's SP 800-61r3 both push the same habit: measure the phases, then improve the slowest one.

The best evaluation method I know needs no live risk at all: replay past incidents. Take the last 30 to 50 postmortems, reconstruct the alert and the telemetry available at the time, run the agent against them, and score its hypotheses against the known root cause. HolmesGPT's community builds evals in a similar spirit. Then run in shadow mode on live incidents for a few weeks, where the agent posts its triage but nobody is required to act on it, and compare against what the humans found.

9. The humans still run the incident

Google's incident management model separates roles: incident commander, operations lead, communications lead and planning lead (SRE book, Managing Incidents). An agent fits naturally into supporting roles: it's an excellent scribe, a fast investigator and a decent first draft of stakeholder updates. It is not the incident commander. Severity, customer communication and "we're done" stay human decisions, and the IC can halt the agent with one command.

Two practical points from working on teams split across time zones. For a team in India and Germany, the handover is where context gets lost, and an agent-maintained timeline is useful there: the next shift reads one coherent summary instead of scrolling a chat. And keep the postmortem blameless and honest about the agent too. If it anchored the team on the wrong hypothesis for 15 minutes, write that down, the same way you would for a misleading dashboard (SRE book, Postmortem Culture).

I already use agentic AI day to day to write and maintain Terraform, and the lesson carries over: the agent is fastest where a human reviews its output before it touches production. Incident response is the same work under time pressure, so the review loop has to be faster, not absent.

10. A 90-day rollout plan

The checklist before any agent gets near the pager:

1. Humans are paged in parallel, never through the agent.
2. The agent's Kubernetes and cloud identities can't read Secrets, exec into pods or write anything.
3. Every tool call goes through a gateway and lands in an audit log and the incident timeline.
4. Actions come only from a reviewed catalogue, with preconditions, verification and rollback.
5. An OPA (or equivalent) policy outside the model decides auto, needs_approval or deny, and it's tested in CI.
6. Auto actions are rate-limited per incident, and a harmful action demotes that action type automatically.
7. Every claim the agent makes links to the query and the raw result.
8. You've replayed past incidents and know the agent's hit rate before it goes live.
9. Change freezes and error budget policy apply to the agent exactly as they apply to people.
10. You run game days with the agent off, so the team keeps its skills.

The short version

AI SRE agents are worth adopting, mostly for the first ten minutes: reading everything at once, ranking hypotheses and writing things down. The risk sits almost entirely in acting. So separate the two. Give the agent read-only eyes everywhere, make it propose actions from a catalogue you wrote, put a policy engine and a human between the proposal and production, and let it earn more authority one action type at a time. That's not a brake on AI in operations. It's how you get to use it at all in a production environment.