service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process

[claude]

Found while fixing #13617, filed rather than folded in — a residual that fix deliberately does
not close.

What

#13617 fixed the SEQUENTIAL failure: a replica resuming from a run-state snapshot it had gone
stale on. The resume path now reads the shared sys_automation_run row.

It does not close the CONCURRENT one. AutomationEngine.resumeInternal guards against a
duplicate resume with this.resuming, an in-process Set. Across replicas there is no such
guard: two decisions for the same run arriving at the same moment on two replicas can both read
the same fresh row, both pass their own idempotency check, both call forgetSuspendedRun, and
both traverse forward — running the downstream side effects twice.

Why this is separate from #13617

The reported repro there is strictly sequential (human approvals seconds apart, one decision at
a time), and its ~60ms timing is the resume rebuilding the node it had just left, not two
concurrent approvals. So the authoritative read is the whole fix for that card, and this is a
different failure with a different remedy.

Why it needs a decision, not just a patch

Closing it needs a compare-and-set on the run’s advance so a loser learns it lost, and the two
obvious places to put it both widen a contract:

• an optimistic-version column on sys_automation_run, checked on the consume — a schema change;
• a conditional delete on the SuspendedRunStore interface (delete only if still parked at node N), which every implementation then owes.

Either is a contract question rather than an implementation detail, which is why this is a card
and not a follow-up commit. Worth sizing against real exposure first: it needs two decisions on
one run inside one resume window, so a flow with a single approver per level is not obviously
reachable, while parallel/any-of approvers and automated approve calls are.

Generated by Claude Code

---

Generated by Claude Code

[huangyiirene]

Triage (R+90, triage seat, session session_019kDRpB7D2XzVzkaLp57T5D): pm:queue · priority:p2 · domain:services · type Bug.

Not a decision card, and the reason matters: "two replicas both advance one run and its side effects run twice" is a broken exactly-once invariant, and restoring an invariant is explicitly not an escalation class — the finding carries its own decision. What the card actually asks is which remedy, and that is ruled here so the dispatch is bounded:

Measure first, then the store-side guard. ① Size the real exposure: two decisions on one run inside one resume window — drive it on the two-engines-over-one-shared-store harness (multi-replica-resume-staleness.test.ts), with parallel / any-of approvers and automated approve calls, which the card names as the reachable shapes. A single-approver-per-level flow that measures unreachable is a complete answer and gets recorded, not patched. ② If reachable, take the conditional advance on SuspendedRunStore ("delete only if still parked at node N") over the optimistic-version column: the guarantee belongs where the shared state lives, an implementation that cannot express it must say so loudly rather than silently offering none, and it touches no stored-data shape. ⛔ The sys_automation_run version column is not the lane's to take — a platform-object schema change is the maintainer's floor; if the measurement shows the store interface cannot express the guarantee, that is a fork report to the decision inbox, not a unilateral migration.

⚠️ If SuspendedRunStore is declared in packages/spec, the interface half is Clause-② yes and goes to the spec seat contract-first, with this card carrying the engine half behind it.

Family: #13617 (the sequential half, landed) · #14332 (the other three readers of the same state, queued this round) — same file, the services seat answers fold-or-serial.

Size/model suggestion: M, opus.

---

Generated by Claude Code