An agent that can explain an incident is useful. Giving it production access adds another question: what, exactly, is it allowed to change when its explanation might be wrong?

A proposal arrives during an incident

Consider a fictional checkout service. Every timestamp, measurement, and configuration below is invented for this worked example; none describes a real deployment or measured agent performance. The scenario extends the Lab's rollback exercise.

At 14:12 UTC, an assistant recommends reverting checkout version 2.8. Its explanation sounds plausible: a recent deployment, followed by slow requests. The evidence packet says something different:

  • 13:59: the inventory client configuration changed its maximum total attempts per request from one to three.
  • 14:02: checkout p95 latency rose from 280 milliseconds to 2.4 seconds. The inventory queue began growing.
  • 14:08: checkout version 2.8 was deployed. Instances still running 2.7 also showed elevated latency.
  • 14:03–14:11: sampled traces showed extra time waiting on inventory; sampled database spans stayed near their earlier duration. Inventory calls per checkout request were roughly three times their earlier level.

The deployment cannot explain symptoms that began six minutes before it. That rejects the assistant's stated rationale; it does not prove the deployment has no additional effect. Likewise, a small trace sample does not clear the database of every possible problem.

Retry amplification is a hypothesis worth testing. Google's overload chapter explains how retries can increase load and describes limiting attempts and aggregate retry traffic. That mechanism fits this packet, but the source cannot establish the cause of our invented incident. Google SRE: Handling Overload.

The next useful answer is a bounded investigation, not an immediate write.

Let the assistant gather a reviewable packet

For this example, I would initially grant access only to approved telemetry views and configuration history for checkout and inventory. Limit queries to the incident window, cap query cost and returned records, and use credentials that cannot write. Read access still needs boundaries around sensitive fields and expensive queries.

Ask for the query, time range, capture time, and result behind each material claim. “Inventory is overloaded” is too broad. “The inventory queue grew during 14:02–14:12, using this query and these service labels” is something a reviewer can inspect.

The packet should request missing evidence: whether attempt counts include initial calls, whether telemetry is delayed, whether traffic changed, and which instances received the configuration. In this scenario, a staging replay using the previous setting avoids queue growth. That strengthens the hypothesis; staging does not establish what will happen in production.

Keep the alternative explanations attached to the proposal. A dependency slowdown might have triggered retries that then amplified the problem. Reducing attempts could lower pressure while leaving the original fault unresolved. It could also expose more failures to callers.

Make approval meaningful

Suppose the inventory owner is ready to consider a limited test. Here is the approval record I would want for this fictional system. The values illustrate a contract; they are not production defaults.

  • Target and change: on checkout-canary-01 only, change the inventory client's maximum total attempts from three to one. Preserve timeouts, routing, and every other instance.
  • Preconditions: confirm the exact instance identity, current configuration revision, current value of three, and absence of a concurrent rollout. Attach telemetry ending within the last minute. Otherwise stop for review.
  • Authority and lifetime: record the authorized owner's identity and approval of this exact proposal. Permit one application before 14:20 UTC; an expired or changed proposal needs another decision.
  • Expected effect: fewer inventory attempts from this instance, with less waiting if retries are contributing. Increased user-visible failures remain a stated risk.
  • Observation: compare its request errors, latency, and attempts with unchanged instances and inspect the shared inventory queue for five minutes. The operator owns the review.
  • Stop and recovery: in this example, sustained canary errors above 1% for a full minute, missing telemetry, or unexpected configuration changes stop further changes and trigger operator review. No fleet rollout is authorized.

The recovery note must name a possible action without pretending it is harmless. Restoring three attempts might recover transient requests but increase load again. This contract requires a fresh operator decision before that restoration; stopping the experiment does not silently authorize another write.

Canary comparisons also need care. Google's canary chapter discusses representative traffic and shared dependencies: a changed instance can influence its controls. Here, the shared inventory queue makes those controls imperfect, and a five-minute window may have too little traffic for a conclusion. Google SRE: Canarying Releases.

Enforce the boundary outside the model

Approval should produce a specific record that a separate executor can validate. In the design I would test, the assistant proposes a typed operation; it never receives a general shell credential. The executor checks the approved target, allowed field and value, expiry, approver, and expected configuration revision. It claims the single-use approval atomically before attempting the write, preventing concurrent or repeated use.

Those checks need deterministic enforcement. Asking the model to remember its limits leaves the same component proposing the action and policing it. A changed target or argument invalidates the approval. Capturing the decision and the actual result makes later review possible.

The revision check must be enforced when the configuration is written, using a conditional update or equivalent concurrency control. A read followed by an unconditional write leaves a race. If the platform cannot enforce the precondition, this proposal is not ready for unattended execution.

An expired approval authorizes no new action. A timeout authorizes no automatic retry. First read back the state and reconcile the result. AWS's retry guidance notes that a failed or timed-out request may already have produced side effects. An unknown outcome deserves investigation, even when a retry looks convenient. AWS: Timeouts, retries, and backoff with jitter.

These are proposed controls for a system to build and test. This site does not implement such an executor or connect to production.

Watch the service after the action

A successful configuration response proves little about user experience. Record when the value actually takes effect, whether traffic reached the canary, and whether the evidence is fresh enough to evaluate the change.

Compare equivalent request types and record their counts. A quiet canary with no errors may simply have handled almost no requests. If latency improves while errors worsen, report both. If the inventory queue improves across every instance, the canary's contribution remains uncertain.

Keep observing after an approval expires: expiry ends permission to act, not responsibility for an action already taken. If telemetry disappears, record the outcome as unknown and hand control back to the operator. No recovery result is claimed in this example. Even a configuration reversal cannot undo requests that have already failed.

Evaluate the whole decision

Before expanding access, replay packets like this with missing data, contradictory timestamps, delayed metrics, concurrent changes, and ambiguous write outcomes. Compare an assisted workflow with a human baseline using the same evidence. Count review and correction time as part of the work.

  • Can each important claim be traced to an observation?
  • Does contradictory evidence change the recommendation?
  • Can the reviewer identify the exact approved operation and its expiry?
  • Does the executor reject stale state, altered arguments, and reused approvals?
  • Does an unknown outcome stop further writes?
  • Can another operator continue from the recorded evidence without reconstructing the incident?

These are evaluation criteria, not results. I would expand authority only after the workflow demonstrates useful decisions within its existing boundary. The first capability worth building may be an assistant that notices the timeline is wrong and makes the next investigation easier to review.