From b1855bd84a4e67c58812b98ff1c53650bb86b99c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 17:53:03 +0000 Subject: [PATCH 1/4] fix(metadata-protocol): bump write epoch on remote metadata mutation receipt applyRemoteMetadataMutation converged a peer replica's registry on a peer's metadata.mutated cluster signal but never moved this replica's write epoch, so meta-overlay-cache's row set stayed "fresh" for the rest of its TTL. A read landing inside that residue window re-hydrated the just-deleted row straight back into the registry the bridge had just healed, converting a bounded ~30s residue into an unbounded one. Adds bumpWriteEpoch (meta-overlay-cache.ts), a structural sibling to readWriteEpoch that retires the cache without importing @objectstack/objectql, and calls it from applyRemoteMetadataMutation after registry convergence and before notifyMutationListenersLocal (#5109 invalidate-before-notify rule). Mirrors authz-invalidation-bridge.ts's epoch.bump('remote') on the identical substrate. Updates the three pins ruling A' authorizes in protocol.datasource-delete-prolongation.test.ts (two UNBOUNDED arms invert to bounded-at-0ms; the SCOPED-kernel arm's bound moves from TTL_MS to 0, with its rationale rewritten), plus unit coverage for bumpWriteEpoch in meta-overlay-cache.test.ts. Fixes #13609 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- ...ata-protocol-remote-mutation-epoch-bump.md | 35 ++++ .../src/meta-overlay-cache.test.ts | 48 ++++++ .../src/meta-overlay-cache.ts | 77 +++++++-- ...col.datasource-delete-prolongation.test.ts | 159 +++++++++++------- packages/metadata-protocol/src/protocol.ts | 18 ++ 5 files changed, 269 insertions(+), 68 deletions(-) create mode 100644 .changeset/metadata-protocol-remote-mutation-epoch-bump.md diff --git a/.changeset/metadata-protocol-remote-mutation-epoch-bump.md b/.changeset/metadata-protocol-remote-mutation-epoch-bump.md new file mode 100644 index 0000000000..5516766c24 --- /dev/null +++ b/.changeset/metadata-protocol-remote-mutation-epoch-bump.md @@ -0,0 +1,35 @@ +--- +'@objectstack/metadata-protocol': patch +--- + +Fix: a peer replica's `meta-overlay-cache` no longer re-serves a datasource (or +any overlay row) a cluster peer just deleted + +`applyRemoteMetadataMutation` — the receipt side of the `metadata.mutated` +cluster channel (#13331) — converged a peer replica's in-memory registry +correctly, but performed no local engine write of its own, so the peer's +`meta-overlay-cache` write-epoch never moved and its pre-mutation row set +stayed "fresh" for the rest of its TTL (default 30s, +`OS_METADATA_OVERLAY_CACHE_TTL_MS`). A single `GET /api/v1/meta/:type` read of +that replica's own door, landing inside that residue window, then ran +`hydrateOverlayIntoRegistry` over the stale rows and wrote the just-deleted +entry straight back into the registry the bridge had just healed — and the +registry itself carries no TTL, so that one read converted a bounded ~30s +residue into an unbounded one for the rest of the process's life. + +`applyRemoteMetadataMutation` now retires this replica's overlay-cache entries +at the moment of convergence — after the registry-convergence branch and +before `notifyMutationListenersLocal` (the #5109 invalidate-before-notify +rule) — via a new structural helper, `bumpWriteEpoch`, declared beside the +existing `readWriteEpoch` in `meta-overlay-cache.ts`. This package must not +import `@objectstack/objectql`, so the bump is spelled the same +feature-detected way `readWriteEpoch` already is, never as a direct import of +the epoch type. The metadata cluster channel now gets the same write-epoch +bump the authorization cluster channel already had +(`authz-invalidation-bridge.ts`'s `epoch.bump('remote')`, on the identical +substrate) — closing an asymmetry between the two, not adding a new mechanism. + +No public API changes: `bumpWriteEpoch` is package-internal (not re-exported +from `src/index.ts`, matching `meta-overlay-cache.ts`'s existing +`metaOverlayCacheEntryCount`), and its one caller is the existing +`applyRemoteMetadataMutation` receipt path. diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index e12dee1ebc..ecedde3abb 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -84,6 +84,7 @@ import { describe, expect, it } from 'vitest'; import { ObjectStackProtocolImplementation } from './protocol.js'; import { + bumpWriteEpoch, META_OVERLAY_CACHE_DEFAULT_TTL_MS, metaOverlayCacheEntryCount, metaOverlayCacheTtlMs, @@ -385,6 +386,53 @@ describe('[#11967] §3 a success is cached ONLY when the engine exposes the writ }); }); +// ═══════════════════════════════════════════════════════════════════════════ +// [#13609] bumpWriteEpoch — the structural sibling that retires this cache +// from OUTSIDE a local engine write, mirroring `authz-invalidation-bridge.ts`'s +// `epoch.bump('remote')` on the identical substrate. `protocol.ts`'s +// `applyRemoteMetadataMutation` is the one call site (see +// `protocol.datasource-delete-prolongation.test.ts` for the end-to-end +// measurement); this block pins the helper itself, the same way §3 above pins +// `readWriteEpoch` apart from any one caller. +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#13609] bumpWriteEpoch — the OUTSIDE-a-write invalidation seam', () => { + it('bumps the seam and returns its new value when the engine exposes one', () => { + const engine = { writeEpoch: makeEpochSeam() }; + expect(readWriteEpoch(engine)).toBe(0); + expect(bumpWriteEpoch(engine, 'remote')).toBe(1); + expect(readWriteEpoch(engine)).toBe(1); + }); + + it('declines the same way readWriteEpoch does — no seam, or only a partial one', () => { + expect(bumpWriteEpoch({ writeEpoch: { current: 3 } }, 'remote')).toBeUndefined(); + expect(bumpWriteEpoch({ writeEpoch: { current: 3, bump: () => 4 } }, 'remote')).toBeUndefined(); + expect(bumpWriteEpoch({}, 'remote')).toBeUndefined(); + expect(bumpWriteEpoch(null, 'remote')).toBeUndefined(); + }); + + it('retires a live cache entry the same way a local engine write does', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); + + await h.protocol.getMetaItems({ type: 'object' }); + const perCall = h.finds.length; + expect(perCall).toBeGreaterThan(0); + + // A repeat still hits — the control half of this assertion, paired per + // this file's own header rule. + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(perCall); + + // The bump this pin is about: never a local `insert`/`update`/`delete` + // on `h.engine` — exactly what a PEER's converged mutation looks like + // from this replica's own engine's point of view. + bumpWriteEpoch(h.engine, 'remote'); + + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(perCall * 2); + }); +}); + // ═══════════════════════════════════════════════════════════════════════════ // 4. Negative caching — the bulk of leg D's win (#11633 §1, §4) // ═══════════════════════════════════════════════════════════════════════════ diff --git a/packages/metadata-protocol/src/meta-overlay-cache.ts b/packages/metadata-protocol/src/meta-overlay-cache.ts index e4339d9612..8e7279aba1 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.ts @@ -72,10 +72,17 @@ * this cache synchronously, in-process, before the next read. * 2. **TTL** — `OS_METADATA_OVERLAY_CACHE_TTL_MS`, default 30s, `0` = off. * The residual bound, covering only what the epoch cannot see: a PEER - * node's write on a deployment with no `authz.invalidated` bridge attached. - * With that bridge, a peer's hint bumps the LOCAL epoch - * (`authz-invalidation-bridge.ts` calls `epoch.bump('remote')`), so - * cross-node convergence narrows for free. + * node's write on a deployment with no cluster bridge attached at all. + * With one attached, the bridge's own receipt path bumps the LOCAL epoch — + * `authz-invalidation-bridge.ts` calls `epoch.bump('remote')` for the + * authorization cache, and `protocol.ts`'s `applyRemoteMetadataMutation` + * calls {@link bumpWriteEpoch}`(engine, 'remote')` for THIS one, after its + * registry converges and before local listeners run (#13609, ruling A′, + * 2026-09-03) — so cross-node convergence narrows for free for both. + * ⚠️ Before that fix, the metadata bridge converged the registry but never + * bumped this cache, so a read landing inside the residue window fed the + * untimed registry a row this cache should already have retired — see + * `protocol.datasource-delete-prolongation.test.ts` for the measurement. * * ⚠️ **`metadata.changed` is NOT a trigger here, and #11633 §4's expectation * that it would be does not survive measurement.** That channel is published by @@ -192,13 +199,29 @@ export function metaOverlayCacheTtlMs( return Math.floor(parsed); } +/** The structural shape read off `engine.writeEpoch`, once validated. */ +interface WriteEpochSeam { + readonly current: number; + bump(reason: string): number; +} + /** - * The engine's current write epoch, or `undefined` when this engine carries no - * such seam. Mirrors `isWriteEpochLike` from `@objectstack/objectql` rather - * than importing it — see the header for why that import is unavailable in this - * direction. + * `engine.writeEpoch`, validated against the full `{ current, bump, subscribe + * }` surface — or `undefined` when the engine carries no such seam, or only a + * partial one. Mirrors `isWriteEpochLike` from `@objectstack/objectql` rather + * than importing it (see this file's header for why that import is + * unavailable in this direction): both {@link readWriteEpoch} and + * {@link bumpWriteEpoch} resolve through here, so the two never drift on what + * counts as a real seam. + * + * ⚠️ The whole surface is checked, `subscribe` included, even though neither + * caller uses it: a `{ current, bump }` pair with no `subscribe` is not this + * engine's real epoch either (see `readWriteEpoch accepts the full surface and + * refuses every partial one` in this package's test file), and checking less + * here would silently let such a shape through one call path and not the + * other. */ -export function readWriteEpoch(engine: unknown): number | undefined { +function resolveWriteEpochSeam(engine: unknown): WriteEpochSeam | undefined { if (!engine || typeof engine !== 'object') return undefined; const epoch = (engine as { writeEpoch?: unknown }).writeEpoch; if (!epoch || typeof epoch !== 'object') return undefined; @@ -210,7 +233,41 @@ export function readWriteEpoch(engine: unknown): number | undefined { ) { return undefined; } - return seam.current; + return seam as unknown as WriteEpochSeam; +} + +/** + * The engine's current write epoch, or `undefined` when this engine carries no + * such seam. + */ +export function readWriteEpoch(engine: unknown): number | undefined { + return resolveWriteEpochSeam(engine)?.current; +} + +/** + * [#13609] Bump the engine's write epoch from OUTSIDE this engine's own write + * path — the structural sibling to {@link readWriteEpoch}, spelled the same + * way and for the same reason: `@objectstack/metadata-protocol` must not + * import `@objectstack/objectql`, so the epoch's real type + * (`WriteEpochLike`/`AuthzInvalidationReason`) is never named here, only its + * shape. + * + * The call this exists for is `protocol.ts`'s `applyRemoteMetadataMutation`, + * mirroring `authz-invalidation-bridge.ts`'s `epoch.bump('remote')` on the + * IDENTICAL substrate (#11968) — that bridge already retires this package's + * sibling cache (the authorization one) the moment a peer's hint converges; + * this is the metadata cluster channel getting the same bump the authz + * channel already had. See this module's header, "Invalidation" ①, for why a + * PEER's mutation does not otherwise move this replica's epoch at all. + * + * A no-op — declining exactly like {@link readWriteEpoch} — when this engine + * exposes no write-epoch seam, or only a partial one. + * + * @returns the epoch's new value, or `undefined` when there was no seam to + * bump. + */ +export function bumpWriteEpoch(engine: unknown, reason: string): number | undefined { + return resolveWriteEpochSeam(engine)?.bump(reason); } /** diff --git a/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts index 2a30331178..b0c6911e5d 100644 --- a/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts +++ b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts @@ -1,11 +1,12 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * [#13609] RE-VERIFICATION — how long does a PEER replica keep serving a - * DELETED datasource on `GET /api/v1/meta/datasource`? + * [#13609] THE FIX — a PEER replica no longer re-serves a DELETED datasource + * on `GET /api/v1/meta/datasource` after its registry converges. * * --------------------------------------------------------------------------- - * What this file is, and why it is a measurement rather than a fix + * What this file is: a re-verification pin that FOUND a second gap, and now + * pins the fix for it in the same file * --------------------------------------------------------------------------- * QA observed a deleted datasource still being served by * `/api/v1/meta/datasource` on all three replicas of a cluster, prolonging an @@ -16,6 +17,18 @@ * re-verification carrier: re-measure the prolongation once that fix lands, and * close only on the measurement. * + * The re-verification (PR #14431) found that #13331 closes the registry-heal + * half but opens a SECOND gap on the read-in-window path (see "Arms" below, + * preserved as history). Ruling A′ (2026-09-03, director batch #22, comment + * 5528370129) ordered the fix for that gap: `applyRemoteMetadataMutation` + * (`protocol.ts`) now retires the peer's `meta-overlay-cache` entry — via + * `bumpWriteEpoch`, structurally, the same substrate + * `authz-invalidation-bridge.ts` already bumps for its own channel — at the + * moment of convergence, before `notifyMutationListenersLocal`. That fix lands + * in the same PR as the three pin changes below. This file is therefore both + * records at once: the measurement that found the gap (kept as history), and + * the pin that now proves it closed. + * * ⭐ The discriminator is DURATION, and it is sharp: * * bounded (≲ one cache TTL window) = fixed @@ -97,41 +110,62 @@ * writes the deleted row straight back into the registry the bridge * just healed — and the registry has no TTL. * - * So the prolongation is neither unconditionally bounded nor unconditionally - * unbounded, and the discriminator is not time but TRAFFIC: + * So the prolongation was — PR #14431's finding, kept as history below — + * neither unconditionally bounded nor unconditionally unbounded, and the + * discriminator was not time but TRAFFIC: + * + * read lands inside the residue window -> UNBOUNDED ("door READ during + * the residue window", THEN) + * no read lands inside it -> BOUNDED, one TTL window + * ("door NOT read during the + * residue window", unchanged) * - * read lands inside the residue window -> UNBOUNDED (cases 4, 6) - * no read lands inside it -> BOUNDED, one TTL window (case 5) + * Both were pinned, side by side, differing in exactly that one step, because + * either reading alone looked like a clean verdict and neither alone was true. * - * Both are pinned, side by side and differing in exactly that one step, because - * either one alone reads as a clean verdict and neither one alone is true. + * ⭐ Ruling A′'s fix collapses the READ-in-window case to bounded-at-0ms too + * (now ✅ below): the traffic discriminator stops mattering once convergence + * itself retires the cache, rather than leaving a TTL a read could still race. * * ⚠️ On a replica actually serving `/api/v1/meta/datasource` — the door QA was * watching — a read inside a 30s window is the ordinary case, not the unlucky * one. The bounded arm is the quiet-replica arm. * * --------------------------------------------------------------------------- - * Why the re-hydration happens, precisely + * Why the re-hydration USED TO happen, precisely (history — PR #14431) * --------------------------------------------------------------------------- - * Nothing on the bridge's receipt path touches the WRITE EPOCH that keys the - * overlay cache. `applyRemoteMetadataMutation` re-reads the row and repairs the - * registry; it performs no engine write, and a peer replica does no writing of - * its own on this path, so the peer's epoch does not move and its pre-delete - * row set stays "fresh" for the rest of its TTL. `getMetaItems` then does what - * the cache's own header says it always does, hit or miss — it runs + * Nothing on the bridge's receipt path touched the WRITE EPOCH that keys the + * overlay cache. `applyRemoteMetadataMutation` re-read the row and repaired the + * registry; it performed no engine write, and a peer replica does no writing of + * its own on this path, so the peer's epoch never moved and its pre-delete row + * set stayed "fresh" for the rest of its TTL. `getMetaItems` then did what the + * cache's own header says it always does, hit or miss — it ran * `hydrateOverlayIntoRegistry` over those rows. * - * ⭐ The comparison that makes this a gap rather than a design: the SIBLING - * bridge for the same substrate does bump it. `authz-invalidation-bridge.ts` + * ⭐ The comparison that made this a gap rather than a design: the SIBLING + * bridge for the same substrate already bumped it. `authz-invalidation-bridge.ts` * calls `epoch.bump('remote')` when it applies a peer hint, and the overlay * cache's own header cites that as the reason cross-node convergence "narrows - * for free" there. The `metadata.mutated` bridge added for #13331 contains no - * epoch reference at all. So the two cross-node paths over one substrate - * disagree, and this door sits on the half that does not invalidate. + * for free" there. The `metadata.mutated` bridge added for #13331 carried no + * epoch reference at all — the two cross-node paths over one substrate + * disagreed, and this door sat on the half that did not invalidate. * - * ⛔ NOT FIXED HERE. This card is a measurement carrier and its source surface - * is read-only, so the reading is reported and the repair is left to a card - * that can be decided on it. + * --------------------------------------------------------------------------- + * ✅ FIXED HERE (ruling A′, 2026-09-03) — why it no longer happens + * --------------------------------------------------------------------------- + * `applyRemoteMetadataMutation` now calls `bumpWriteEpoch(this.engine, + * 'remote')` — the structural sibling `meta-overlay-cache.ts` declares beside + * `readWriteEpoch`, never a direct `@objectstack/objectql` import — after the + * registry-convergence branch and BEFORE `notifyMutationListenersLocal` (the + * #5109 invalidate-before-notify rule that method's own docblock states: a + * listener that re-reads must not observe the event and the pre-event registry + * at the same time — and the overlay cache is exactly such a re-read). The + * bump retires every entry this replica's overlay cache holds in the SAME + * synchronous step that heals the registry, so the read that follows finds a + * cold cache, re-reads `sys_metadata` fresh (empty, for a deleted row), and + * hydrates nothing back in. The metadata channel now gets the write-epoch bump + * the authz channel already had — the asymmetry above is closed, not routed + * around. * * --------------------------------------------------------------------------- * Four-seam checklist (the card's own elimination list) — verdicts @@ -144,11 +178,14 @@ * boot-only, and it reads the already-corrected DB, so it cannot explain a * steady-state cross-replica prolongation without a restart. Cited, not * re-derived (the dispatch forbids re-deriving that finding). - * 3. list-cache TTL (#5109) ............. ⛔ NOT the bounded residue the ruling - * expected. At this door the cache is `meta-overlay-cache`, not - * `MetadataManager.listCache` — and it does not merely delay the correct - * answer, it FEEDS the untimed registry, converting a 30s residue into an - * unbounded one on any read. That conversion is what this file measures. + * 3. list-cache TTL (#5109) ............. RE-MEASURED, then CLOSED. At this + * door the cache is `meta-overlay-cache`, not `MetadataManager.listCache` — + * PR #14431 found it did not merely delay the correct answer, it FED the + * untimed registry, converting a 30s residue into an unbounded one on any + * read landing in the window. Ruling A′'s bump on the receipt path retires + * this cache at the moment of convergence, so the read that follows has + * nothing stale left to feed the registry with. This file measures both + * states, before and after. * 4. same class as #13578 ............... ELIMINATED by PR #13883 — same * symptom, opposite mechanism (that driver registry had NO eviction door; * this one's door exists and does broadcast). Cited, not re-derived. @@ -622,7 +659,7 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving expect(cluster.peer.removedEntries).toContain('datasource|billing_db'); }); - it('⛔ …but the peer’s very next READ re-hydrates the deleted row from its own stale overlay cache', async () => { + it('✅ [FIXED] …and the peer’s very next READ no longer re-hydrates the deleted row', async () => { const cluster = makeCluster({ attach: true }); await seedServedDatasources(cluster, ['billing_db']); @@ -631,21 +668,21 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving expect(cluster.peer.registry.listItems('datasource')).toEqual([]); // One read of the peer's own door, with the clock untouched. - expect(await serves(cluster.peer, 'billing_db')).toBe(true); - - // ⭐ THE SEAM. The convergence retired the registry entry but did NOT - // retire the peer's overlay-cache entry — nothing on the receipt path - // touches the write epoch that keys it, and the peer does no writing of - // its own, so the pre-delete row set is still "fresh". `getMetaItems` - // then does what its cache's own header says it always does, hit or - // miss: it runs `hydrateOverlayIntoRegistry` over those rows. The - // deleted row is written straight back into the registry the bridge - // just cleaned — and the registry has no TTL. + expect(await serves(cluster.peer, 'billing_db')).toBe(false); + + // ⭐ THE FIX (ruling A′, #13609). `applyRemoteMetadataMutation` now + // bumps the peer's write epoch in the SAME synchronous step that + // retires the registry entry, before `notifyMutationListenersLocal` + // runs — so the peer's overlay-cache entry (keyed on that epoch) is + // ALREADY stale by the time this read arrives, clock untouched or not. + // `getMetaItems` misses the cache, re-reads `sys_metadata` fresh (no + // active row), and hydrates nothing back in — the registry the bridge + // just cleaned stays clean. expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) - .toEqual(['billing_db']); + .toEqual([]); }); - it('⛔ Arm B measured, door READ during the residue window: UNBOUNDED, past 10 windows', async () => { + it('✅ [FIXED] Arm B measured, door READ during the residue window: BOUNDED AT 0 ms', async () => { const cluster = makeCluster({ attach: true }); await seedServedDatasources(cluster, ['billing_db']); @@ -653,14 +690,16 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving await settle(); // `measureProlongationMs` reads the door once BEFORE advancing the - // clock — i.e. inside the residue window, which is what a replica under - // load does continuously. That read converts the bounded cache residue - // into an unbounded registry entry, so waiting the TTL out no longer - // helps: once the cache lapses the registry is the only source left, - // and it is the one now holding the deleted row. + // clock — i.e. inside the residue window, which is what a replica + // under load does continuously. Previously (PR #14431) that read + // converted the bounded cache residue into an unbounded registry + // entry. Now the convergence bump has already retired the cache + // before this first read runs, so the read-in-window traffic + // discriminator this file's header describes stops mattering: there + // is no stale entry left for a read to convert into anything. const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); - expect(prolongation).toBeNull(); - expect(await serves(cluster.peer, 'billing_db')).toBe(true); + expect(prolongation).toBe(0); + expect(await serves(cluster.peer, 'billing_db')).toBe(false); }); it('⭐ Arm B measured, door NOT read during the residue window: BOUNDED by one TTL window', async () => { @@ -708,29 +747,33 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving expect(prolongation).toBeNull(); }); - it('⭐ SCOPED kernel: the same delete IS bounded — at exactly one overlay-cache TTL window', async () => { + it('✅ [FIXED, pin amended by ruling A′] SCOPED kernel: the same delete now converges IMMEDIATELY — 0 ms, not one TTL window', async () => { const cluster = makeCluster({ attach: true, environmentId: 'env_prod' }); await seedServedDatasources(cluster, ['billing_db']); // On a scoped kernel BOTH hydration seams are gated off — the read-side // loop and the write-through alike — so no replica ever holds a local // registry copy of an overlay row, and the only local source is the - // overlay cache, which does expire. + // overlay cache. expect(cluster.peer.registry.listItems('datasource')).toEqual([]); const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); expect(res.success).toBe(true); await settle(); - // ⭐ The bound, stated as a number: one window, and the constant is - // imported rather than retyped so this tracks whatever ships. + // ⭐ PIN AMENDED (ruling A′, 2026-09-03, comment 5528370129). This arm's + // bound was `TTL_MS` (30 000 ms) before the fix — on a scoped kernel the + // overlay cache is the ONLY local source, so its TTL was the one + // residue the original ruling expected to survive. It does not survive: + // the convergence bump (`bumpWriteEpoch`, this PR's fix) retires that + // same overlay cache at the moment of convergence instead of letting it + // lapse on its own clock, so the one local source a scoped kernel has + // is already gone before the first read, not merely bounded by a timer + // a read could still race. The pin keeps asserting a literal number, + // per the ruling — moved to `0`, never loosened to `<=`. const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); - expect(prolongation).toBe(TTL_MS); - expect(TTL_MS).toBe(30_000); + expect(prolongation).toBe(0); - // This is the residue the ruling permits ("listCache TTL is the one - // bounded residue that may legitimately remain") — refined to the cache - // that actually holds it at this door. expect(await serves(cluster.peer, 'billing_db')).toBe(false); }); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index c74a19e0f5..83642a5f8d 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -30,6 +30,7 @@ import { ensureMetadataOverlayIndexes } from './migrations/overlay-index.js'; import { driverCanRunSql, resolveDriverExec } from './migrations/driver-exec.js'; import { SysMetadataRepository, type SysMetadataEngine } from './sys-metadata-repository.js'; import { + bumpWriteEpoch, metaOverlayCacheTtlMs, readMetaOverlayCache, readWriteEpoch, @@ -5169,6 +5170,19 @@ export class ObjectStackProtocolImplementation implements * first, listeners second — the #5109 invalidate-before-notify rule: a * listener that re-reads must not observe the event and the pre-event * registry at the same time. + * + * [#13609] The convergence above heals THIS replica's registry, but on its + * own leaves this replica's `meta-overlay-cache` row set stamped at the + * PRE-mutation write epoch — nothing above performs a local engine write, + * so nothing retires it. A read of `getMetaItems` landing before that + * cache's TTL lapses then re-hydrates the very row this method just healed + * the registry of (measured: `protocol.datasource-delete-prolongation.test.ts`). + * `bumpWriteEpoch` closes that gap the same way + * `authz-invalidation-bridge.ts` already closes it for the authorization + * cache on the identical substrate (#11968) — called here, AFTER + * convergence and BEFORE {@link notifyMutationListenersLocal}, so a + * listener that re-reads through `getMetaItems` sees a cold cache too, not + * only a healed registry. */ private async applyRemoteMetadataMutation(evt: MetadataMutationEvent): Promise { const type = canonicalMetaType(evt.type); @@ -5195,6 +5209,10 @@ export class ObjectStackProtocolImplementation implements } else { await this.restoreArtifactRegistryView(type, evt.name, orgId); } + // [#13609] Retire this replica's overlay-row cache at the moment of + // convergence — never a direct `@objectstack/objectql` import, see + // `bumpWriteEpoch`'s header in `meta-overlay-cache.ts`. + bumpWriteEpoch(this.engine, 'remote'); this.notifyMutationListenersLocal({ ...evt, type }); } From 28cc78412dc1a235b6086983a0367a6194685425 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 18:01:06 +0000 Subject: [PATCH 2/4] docs: tidy @link markup in bumpWriteEpoch cross-reference Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- packages/metadata-protocol/src/meta-overlay-cache.ts | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/packages/metadata-protocol/src/meta-overlay-cache.ts b/packages/metadata-protocol/src/meta-overlay-cache.ts index 8e7279aba1..213f1acf2e 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.ts @@ -76,9 +76,9 @@ * With one attached, the bridge's own receipt path bumps the LOCAL epoch — * `authz-invalidation-bridge.ts` calls `epoch.bump('remote')` for the * authorization cache, and `protocol.ts`'s `applyRemoteMetadataMutation` - * calls {@link bumpWriteEpoch}`(engine, 'remote')` for THIS one, after its - * registry converges and before local listeners run (#13609, ruling A′, - * 2026-09-03) — so cross-node convergence narrows for free for both. + * calls {@link bumpWriteEpoch} for THIS one, after its registry converges + * and before local listeners run (#13609, ruling A′, 2026-09-03) — so + * cross-node convergence narrows for free for both. * ⚠️ Before that fix, the metadata bridge converged the registry but never * bumped this cache, so a read landing inside the residue window fed the * untimed registry a row this cache should already have retired — see From ca9728a8a8cac05261feac4b1dbaab28e576f392 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 18:16:11 +0000 Subject: [PATCH 3/4] chore: re-anchor system-context census after protocol.ts import line shift pnpm check:system-context-census --fix, following our bumpWriteEpoch import addition in protocol.ts (row 21 shifted 1746 -> 1747). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- content/docs/permissions/system-context.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/permissions/system-context.mdx b/content/docs/permissions/system-context.mdx index 0a598b835c..b1c7cae873 100644 --- a/content/docs/permissions/system-context.mdx +++ b/content/docs/permissions/system-context.mdx @@ -112,7 +112,7 @@ that silently does not happen. | 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:11290` | | 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:11473` | | 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:10025` | -| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1746` | +| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1747` | | 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:10073`, `readonly-strict-errors.ts:66` | | 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5892` | | 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3736`, `:3746`, `:3773` | From 8d1f11e7f828feb27341b2575ec021ef27a70f47 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 20:27:05 +0000 Subject: [PATCH 4/4] docs(metadata): the metadata.mutated receipt now retires the overlay-read cache too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #13609's bumpWriteEpoch call in applyRemoteMetadataMutation retires the sys_metadata overlay-row cache directly, on the same shared engine write epoch the authz.invalidated bridge already bumps. Two doc claims dated from before that fix and named authz.invalidated as the only non-local retirement path, with OS_METADATA_OVERLAY_CACHE_TTL_MS as the sole floor otherwise: - content/docs/concepts/metadata-lifecycle.mdx's cross-replica-sync note - content/docs/deployment/environment-variables.mdx's OS_METADATA_OVERLAY_CACHE_TTL_MS and OS_LOCALIZATION_CACHE_TTL_MS rows (the localization cache in packages/core/src/security/resolve-authz-context.ts reads the identical engine.writeEpoch seam, so it is retired by the same bump) Narrowed the lag claim to match: the TTL is now the floor only on a deployment with no cluster bridge attached at all (in-process `memory` driver), not on every deployment lacking the authz.invalidated bridge specifically. content/docs/kernel/cluster.mdx was flagged by the PR's own docs-drift-check bot (it names applyRemoteMetadataMutation) but makes no claim about the overlay-cache/write-epoch bound at all — left unchanged. No code, test, or changeset touched — the patch changeset already on this branch covers the behavior change; this is a documentation-only correction on top of it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- content/docs/concepts/metadata-lifecycle.mdx | 16 +++++++++++----- .../docs/deployment/environment-variables.mdx | 4 ++-- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/content/docs/concepts/metadata-lifecycle.mdx b/content/docs/concepts/metadata-lifecycle.mdx index 36cfd15bbb..f3f0a081a4 100644 --- a/content/docs/concepts/metadata-lifecycle.mdx +++ b/content/docs/concepts/metadata-lifecycle.mdx @@ -190,11 +190,17 @@ The hash is `sha256:` + 64-hex of a canonical (sorted-keys, no-undefined) JSON s > before its own middleware chain runs, so read-your-writes is exact on the node > that made the write. A **peer's** write does not: `metadata.changed` invalidates > the MetadataManager caches this note is about, but it does not retire the -> overlay-read cache. What retires that on a peer is the `authz.invalidated` -> channel — a hint from another node bumps the local write epoch — or, failing -> that, `OS_METADATA_OVERLAY_CACHE_TTL_MS` (default 30s, `0` disables the cache -> outright). So on a deployment with no distributed cluster driver attached, a -> peer's overlay re-read can lag a remote publish by up to that TTL. See +> overlay-read cache. What retires that on a peer is either of two receipt +> paths that bump the same local write epoch: the `metadata.mutated` +> channel's own — `protocol.ts`'s `applyRemoteMetadataMutation` calls +> `bumpWriteEpoch(engine, 'remote')` right after its registry converges and +> before local listeners run (#13609, 2026-09-03) — or the `authz.invalidated` +> channel's bridge, which bumps the same epoch for the authorization cache +> and retires this one too, as a side effect of sharing it. Failing both, the +> floor is `OS_METADATA_OVERLAY_CACHE_TTL_MS` (default 30s, `0` disables the cache +> outright). So only on a deployment with no cluster bridge attached at all — +> the in-process `memory` driver, with no distributed driver behind it — does +> a peer's overlay re-read still lag a remote publish by up to that TTL. See > [Environment variables](/docs/deployment/environment-variables). --- diff --git a/content/docs/deployment/environment-variables.mdx b/content/docs/deployment/environment-variables.mdx index a7939d36c9..a9b8096114 100644 --- a/content/docs/deployment/environment-variables.mdx +++ b/content/docs/deployment/environment-variables.mdx @@ -361,8 +361,8 @@ the hosted ObjectOS Cloud control plane. | `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default **CPU-time** budget for a sandboxed **hook** body (QuickJS, ADR-0102): how much *VM-active* time a body may burn — idle host-await time and a nested hook's own run are NOT charged. A loaded/slow host rarely needs to raise this now (it is not wall-clock), but the knob remains. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. | | `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default **CPU-time** budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). | | `OS_SANDBOX_WALL_CEILING_MS` | number | `30000` | Wall-clock ceiling (ADR-0102) — the backstop that cuts a hook/action body stuck on a host call that never settles (which burns no CPU, so the CPU budget alone would never fire). The effective ceiling is `max(this, cpuBudget)`, so it can never cut a body still inside its CPU budget. Positive integer only; unset keeps 30s. | -| `OS_LOCALIZATION_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of a workspace's reference localization (`timezone` / `locale` / `currency`, read from `sys_setting`) — leg C of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Unlike `OS_AUTHZ_GRANTS_CACHE_TTL_MS` (which is off by default) this one ships **on**, because its invalidation is synchronous and in-process rather than TTL-bound: a `localization` settings change and any engine write both retire a cached answer immediately, so the TTL only bounds what neither seam can see — a write made on another replica with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off), the opposite arm from the grants variable and deliberately so: there `0` is also the default, whereas here folding `3OOO` (letter O) into the default would hand you a **longer** window than the one you were setting. Deployment config only — never a settings row, because `sys_setting` is the table this cache caches. | -| `OS_METADATA_OVERLAY_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of the `sys_metadata` overlay read inside `getMetaItems` — leg D of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Ships **on**, for the same reason as `OS_LOCALIZATION_CACHE_TTL_MS`: invalidation is synchronous and in-process, because every `sys_metadata` write goes through the engine and so advances the write epoch that retires the entry before the next read. What is cached is the overlay ROW SET only — never the merged answer — so the SchemaRegistry, the MetadataService and the artifact table are re-consulted on every call, cached or not, and the read-side registry hydration keeps running on a cache hit. The TTL therefore bounds one thing: a write made on **another replica** with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off) — same arm and same reason as `OS_LOCALIZATION_CACHE_TTL_MS`. Deployment config only, never a settings row. | +| `OS_LOCALIZATION_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of a workspace's reference localization (`timezone` / `locale` / `currency`, read from `sys_setting`) — leg C of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Unlike `OS_AUTHZ_GRANTS_CACHE_TTL_MS` (which is off by default) this one ships **on**, because its invalidation is synchronous and in-process rather than TTL-bound: a `localization` settings change and any engine write both retire a cached answer immediately, so the TTL only bounds what neither seam can see — a write made on another replica with no cluster bridge attached at all: neither the `metadata.mutated` channel's receipt path (`applyRemoteMetadataMutation` calls `bumpWriteEpoch`, #13609) nor the `authz.invalidated` bridge, both of which bump this same epoch when attached. ⚠️ A malformed value reads as `0` (off), the opposite arm from the grants variable and deliberately so: there `0` is also the default, whereas here folding `3OOO` (letter O) into the default would hand you a **longer** window than the one you were setting. Deployment config only — never a settings row, because `sys_setting` is the table this cache caches. | +| `OS_METADATA_OVERLAY_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of the `sys_metadata` overlay read inside `getMetaItems` — leg D of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Ships **on**, for the same reason as `OS_LOCALIZATION_CACHE_TTL_MS`: invalidation is synchronous and in-process, because every `sys_metadata` write goes through the engine and so advances the write epoch that retires the entry before the next read. What is cached is the overlay ROW SET only — never the merged answer — so the SchemaRegistry, the MetadataService and the artifact table are re-consulted on every call, cached or not, and the read-side registry hydration keeps running on a cache hit. The TTL therefore bounds one thing: a write made on **another replica** with no cluster bridge attached at all — neither the `metadata.mutated` channel's own receipt path (`applyRemoteMetadataMutation` calls `bumpWriteEpoch` right after registry convergence, #13609) nor the `authz.invalidated` bridge, either of which retires this entry the moment it is attached and fires. ⚠️ A malformed value reads as `0` (off) — same arm and same reason as `OS_LOCALIZATION_CACHE_TTL_MS`. Deployment config only, never a settings row. | | `OS_INLINE_SEED_BUDGET_MS` | number | `8000` | Time budget for synchronous seed execution at boot before deferring to a worker. | | `OS_TENANT_AUDIT` | flag | `1` | Set to `0` to silence the tenant-isolation audit warnings emitted by the SQL driver. |