02 — Anonymised delivery

A mainland internet company · data-platform operations

Agent for P0 scheduling and data sync

Let an agent clear the path before morning starts.

The platform carried 180 PiB-class Hive warehouse and lakehouse offline pipelines with 60,000+ job instances a day. Backfills, datacenter moves and heterogeneous engines created resource spikes; operators needed more than dashboards when T+1 drifted or a sync job failed.

We put an operations-agent loop on the production substrate: task and DAG state, logs, SQL, lineage, alerts and retry controls become tools. The agent can correlate evidence, explain a likely cause, recommend rerun or routing, and hand write actions to an approval gate. The underlying platform still provides the reliable execution boundary.

In the work

  • A production tool surface

    ZooKeeper, MySQL, rebuildable queues and K8s Jobs expose durable state and safe execution primitives to the agent.

  • Reason over hybrid compute

    Cluster load, queue pressure and retry history inform routing between YARN and Serverless Spark; 8,000+ instances were offloaded on peak days.

  • Diagnose heterogeneous sync

    Connector metadata, CDC state and Paimon tables give the agent evidence across 10+ sources and 100+ production tables.

The problem

A mainland internet company ran large offline pipelines across multiple engines with deep dependencies. T+1 could slip past noon after failures or backfills, while operators still had to diagnose heterogeneous sync by hand without data loss.

Measured result

The execution platform reached 99.9% SLA with a 99.7% monthly baseline success rate; core T+1 readiness moved from frequent 12+ hour delays to before 08:00 daily. The agent-enabled operations loop brought typical diagnosis from 30+ minutes to under 5 minutes, while the wider platform recorded zero P0/P1 incidents since 2023, 10,000+ migrated sync jobs with zero data loss, and alerts falling from thousands per day to hundreds.

Scope

  • P0 scheduling and sync substrate: control plane, execution plane and DAG dependencies
  • Agent tools for task state, logs, SQL, lineage, routing and retry with approval gates
  • 10+ heterogeneous sources, K8s execution, observability and cross-datacenter repair

What shipped

  • → An operations-agent loop: inspect → correlate → explain → recommend → approve → act
  • → Two dual-datacenter migrations covering 10,000+ sync jobs
  • → Flink CDC → Apache Paimon streaming-lakehouse evolution with 100+ production tables

Anonymisation

The client, internal system names and business data are anonymised. Scale, SLA, timing and migration outcomes come from that engagement’s records.

Work

Anonymised delivery

HSK 3 learning agent, from zero

Anonymised delivery

DataClaw BI analysis agent

Anonymised delivery

Data-platform Agent runtime and MCP tool plane

Anonymised delivery

Governed knowledge gate for analysis Agents

Anonymised delivery

Alert-to-remediation operations Agent

Anonymised delivery

Agent control plane with guarded actions

Start a conversation