Alert-to-remediation operations Agent

Turn alert → evidence → recovery into one accountable loop.

Platform operations had recurring inspection, alert handling and job troubleshooting across separate control-plane APIs. The useful Agent was not a general chatbot; it was a bounded workflow that could collect the right artifacts, explain what it found and stop at a write boundary.

We delivered scheduled inspection and alert-driven diagnosis. The Agent follows the task or instance into logs, SQL and lineage, proposes a rerun or next action, and keeps the on-call engineer in the approval path when the action changes production state.

In the work

Scheduled inspection
Hourly coverage replaces a weekly script ritual and creates a consistent starting point for every incident.
Artifacts, not guesses
Instance state, log artifacts, SQL and lineage are collected before the Agent explains a likely cause.
Human-owned writes
Rerun and tuning actions stay scoped and approval-gated; the Agent accelerates diagnosis without making incidents ownerless.

The problem

Platform engineers spent 30+ minutes locating typical failures, while manual inspection scripts consumed 2–4 hours each week. Alerts, logs and job context were available but not connected into one repeatable workflow.

Measured result

Hourly full-scope inspection replaced 2–4 hours of weekly manual scripts. Typical fault localisation fell from 30+ minutes to under 5 minutes, platform on-call volume dropped 60%+, and 30%+ of task-failure questions closed through self-service for a daily user group of about 60 engineers.

Scope