feat(bounty): score only from the backend public feed, fail closed

[echobt]

Summary

Bounty goes live against the CortexLM/backend public API, and the last way it
could pay miners without one is removed. Stacked on
cursor/four-live-challenges-relock-16b4 (PR #202); no challenge is retuned —
bounty stays at 3000 bps, relearn-mm stays off.

The hole this closes. resolve_scoring_backend() had a third answer,
Sim, selected by BOUNTY_FORCE_SIM, and try_fetch_public_snapshot()
turned a missing BOUNTY_BACKEND_PUBLIC_URL into Ok(None) — a skip. So a
host with no feed could accept bug reports and score them locally, on
adjudications no validator can reproduce. ScoringBackend is now
backend_public | unconfigured, the skip is gone, and every read failure
(unset, blank, transport, non-2xx, unparseable, mid-publish) is an error.

Live wiring. crates/bounty-challenge/src/emit.rs adds BountyEmitter:
fetch the two public routes → derive E from the metagraph at
last_epoch_block → sign one leaf per hotkey in E for the current subnet
epoch → POST /v1/weights/raw. The binary wires it from
BASE_CHAIN_ENDPOINT(S), BASE_NETUID, BASE_CHALLENGE_GATEWAY_ENDPOINT,
and BOUNTY_EMIT_POLL_SECS. Validators are unchanged: they fetch the sealed
bundle and never read the bounty feed.

What fail-closed means here (and what it must not mean)

A tick that cannot read the feed pays nobody: no score is invented and no
local scorer is consulted. It still covers E, with
NoScore(ChallengeInternal) — the code BUNDLE_SPEC §3.3.1 defines as
"challenge-side fault; still must cover the participant". Both halves matter:

• It must not pay. An all-NoScore set burns the 3000 bps to uid 0, the
  honest outcome when nothing was adjudicated.
• It must still cover E. I first implemented this as "emit nothing", then
  checked bundle::seal::assert_participant_completeness: D24 requires a leaf
  per participant for every challenge with emission_share_bps > 0, so an
  uncovered bounty row makes POST /v1/admin/seal answer 409
  incomplete_participant_set for the whole bundle. Silence would have let an
  unconfigured bounty host leave relearn's weights unsealed. The emitter
  therefore runs even with no feed configured; only a missing
  BASE_CHALLENGE_SK_FILE stops it, since a leaf the trust root rejects is not
  weight.

Ingest keeps the stricter answer: POST /v1/reports is 503 with no feed,
storing nothing.

A failed tick also avoids taking back a score: once the process has scored an
epoch, an outage inside that same epoch holds rather than superseding a
champion's leaf with a burn. That watermark is in-process (the gateway exposes
POST /v1/weights/raw with no GET side, so there is nothing to consult), so a
restart during an outage can still burn an epoch that had scores; the next
successful tick supersedes it back. The bias is deliberate — burning pays
nobody who was not already paid, staying silent would 409 everyone.

Anti-farm behaviour from this branch is untouched: title==body and token-thin
bodies are still 400, and a repeated fingerprint (including one already
closed) is still duplicate.

A mixed pair is not a snapshot (Greptile P1, fixed in 66fb79db)

Greptile caught that the first revision read /leaderboard and /reports once
apiece and signed whatever came back. Those are separate GETs, so a publish
landing between them gets combined into one scoring input, and the failure is
not cosmetic: every tally comes from /reports
(score_plan_from_snapshot → holdouts_from_reports), so a stale half can
under-count a miner's valid rows or drop it to NotAttempted, while
/leaderboard sets the champion walk order that judge_challenger consumes.
Either way the epoch carries a verdict the backend never published.

fetch_public_snapshot now re-reads the pair until two consecutive reads
agree, bounded at four reads. Two matching reads bracket the in-between read on
both routes, so a publish anywhere in that window shows up as a mismatch. Three
details worth reviewing:

• It compares the parsed PublicSnapshot, not the raw bodies, so a field
  the public DTO does not model (a generated_at stamp, say) cannot make a
  still feed look like a moving one and burn the epoch forever.
• A settling feed is a retry, not a lost epoch — that is what the bound
  buys, and a_feed_that_settles_is_scored_at_the_revision_it_settled_on
  pins it.
• A feed that never holds still is BackendError::Inconsistent, which falls
  into the existing burn/hold path rather than signing a guess.

This needs nothing from the backend. If it later publishes a revision or ETag
on both routes, the check collapses to a single round.

Deploy artifacts

• docker-compose.yml: BOUNTY_EMIT_POLL_SECS + gateway endpoint on the
  service; BASE_NETUID stays in the env file and chain endpoints in the
  overlays (no value baked over env_file).
• env-staging.yml / env-prod.yml: testnet / mainnet chain failover lists
  for the emitter; BOUNTY_BACKEND_PUBLIC_URL still host-set and empty in git.
• env-local.yml / docker-compose.e2e.yml: sim knob dropped — a laptop with
  no feed now 503s on ingest and emits burn leaves only.
• deploy/env/bounty-challenge.env.example: feed + emitter knobs documented,
  names only.
• assert-compose-matrix.sh: no BOUNTY_FORCE_SIM in any compose file, and no
  baked backend host in staging/prod.
• local-e2e.sh --smoke: probe_bounty_fail_closed POSTs real ingest and
  asserts 503 (no feed) or 401 invalid_session (feed configured), rather
  than trusting /health.

No Modal, no secrets, no backend hostnames in git.

Known gap this PR does not close (out of scope)

relearn, relearn-image, and relearn-agent still have emit_epoch() as a
library function with no emitter loop and no gateway client, so the
four-challenge bundle cannot seal on main until they emit too — bounty is now
the only one of the four that covers its own trust-root row. Not touched here
per "do not retune other challenges", but it is the next thing needed for a
sealed four-challenge bundle.

Greptile

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Greptile reviewed 23606f11 and raised one P1 (torn feed snapshot), fixed in
66fb79db with a regression test — see the section above. Its T-Rex runtime
reproducer could not run ("execution service unavailable"), so the property is
pinned by
a_feed_that_never_holds_still_is_never_signed_as_one_snapshot instead: the
mock feed publishes a new revision on every request, and the emitter must burn
rather than score. That test fails against the pre-fix read.

Test plan

Every step of .github/workflows/ci.yml's ci job was run locally on
66fb79db, using the versions CI pins (Rust 1.96.0, cargo-deny 0.20.2 as
cargo-deny-action@v2 installs):

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace — 299 suites, 1851 passed, 0 failed
• cargo deny check — advisories ok, bans ok, licenses ok, sources ok
• cargo run -p xtask -- loc-cap / consensus-lint / spec-check /
  design-check / external-docs-check
• cargo clippy -p validator-bin --features dcap --all-targets -- -D warnings
• bash deploy/scripts/assert-compose-matrix.sh — all checks passed,
  including the new bounty assertions. This was the one gate the first
  revision could not run; docker compose config needs no daemon, so the
  standalone compose plugin renders the whole role × env matrix.
• bash -n on both edited shell scripts

GitHub Actions does not report on this PR, and that is the trigger config
rather than a failure: ci.yml fires on pull_request: branches: [main], and
this PR targets cursor/four-live-challenges-relock-16b4 (#202). Retargeting
main to make the checks appear would swell the diff to ~199 files of #202 plus
its own parent and duplicate two open PRs, so the base stays put and the gate
matrix above is the evidence. CI will run for real when this lands on a
main-targeting PR after the stack merges.

New tests, stated as the property they defend:

Test	Proves
emit_fail_closed::mock_backend_rows_become_scored_leaves_for_metagraph_hotkeys	A mock backend serving the two public routes yields three leaves for three metagraph hotkeys: champion Score > 0, fabricator InvalidResponse, silent hotkey NotAttempted
…::an_unset_backend_url_burns_without_paying_anyone / a_blank_backend_url_…	Unset (and the empty-compose-variable shape) covers all of E with ChallengeInternal and pays nobody — no score field on any leaf, scored_epoch still 0
…::a_failing_backend_burns_until_it_recovers	A 5xx feed pays nobody, and the recovered tick supersedes the burn with the published score
…::an_outage_after_a_scored_epoch_holds_instead_of_burning_it	An outage inside a scored epoch posts nothing and leaves the champion's score standing
…::a_feed_that_never_holds_still_is_never_signed_as_one_snapshot	A feed publishing a new revision on every request is Inconsistent and burns, rather than signing a mixed pair
…::a_feed_that_settles_is_scored_at_the_revision_it_settled_on	The re-read is a bounded retry, so ordinary backend activity does not cost miners an epoch
backend.rs::an_unset_url_is_an_error_not_a_skip, an_unreachable_backend_is_a_fetch_error	The client itself never returns a skip
bounty-challenge-task::the_retired_sim_opt_in_cannot_turn_scoring_back_on	BOUNTY_FORCE_SIM=1/true/yes still resolves Unconfigured
bounty-http::an_unconfigured_host_refuses_reports_instead_of_collecting_them	Ingest 503s and banks nothing
bounty-http::status_publishes_the_scorer_and_the_off_score_gate	/v1/status no longer advertises force_sim
bins/bounty-challenge boot tests	The emitter is wired even with no feed (so E stays covered), and not wired without a challenge key

Risk

Emission-shaped but bounded. bounty stays at 3000 bps and no other
challenge is retuned. Three things an operator should know:

1. A host that had been running on BOUNTY_FORCE_SIM=1 now refuses ingest and
   pays nobody until BOUNTY_BACKEND_PUBLIC_URL is set. Intended — the sim
   path was paying on unverifiable numbers — but it is a live-behaviour change
   on any host that set it. The boot log says so explicitly.
2. Bounty now posts leaves to the gateway. The bounty_sk public key must
   match config/challenges.toml (d2ffbe70…) or the gateway rejects them;
   local-e2e.sh already aligns that key.
3. A restart while the backend feed is down can burn the current epoch's
   bounty scores (documented above). It errs toward paying nobody, never
   toward paying the wrong hotkey, and self-heals on the next good tick.

Feed load doubles on the happy path (two reads of each route per tick instead
of one) — four small GETs per BOUNTY_EMIT_POLL_SECS, 120s by default.

GET /v1/status drops the force_sim field and scoring_backend can no
longer be sim. Nothing in-repo consumes either.

Naming

I did not rename BASE_* environment variables, deployed host paths
(/opt/base, /run/base, …), GHCR baseintelligence/base package names, or
base-*-v1 cryptographic domain tags, unless this PR’s purpose is a coordinated
cutover documented in docs/NAMING.md.

  

[greptile-apps]

Greptile Summary

The bounty emitter now sources scoring data from the backend public feed, fails closed when the feed cannot be read, and sends signed participant results through the gateway. However, it can sign a leaderboard from one backend publication together with reports from another, producing incorrect tally or ordering decisions.

Confidence Score: 4/5

Do not merge until backend leaderboard and report responses are pinned to the same publication before scores are signed.

One confirmed scoring-correctness failure remains: repeated reads can accept a stable combination of data from different backend publications.

Files Needing Attention: crates/bounty-challenge/src/backend.rs

T-Rex Logs

What T-Rex did

• Ran the stable torn snapshot test in the repository; it exited with code 0 and reported ACCEPTED_MIXED_REVISION leaderboard=A(valid_count=1) reports=B(id=report-from-revision-B) requests=4, and harness source and command output were captured as artifacts.
• Generated a proof for a posted P1 finding (finding-comment-proof) and attached the stable torn snapshot harness source and its command output as evidence.
• Generated an additional proof for another posted P1 finding (finding-comment-proof).

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   Stable torn snapshots pass the consecutive-composite check

   • Bug
     • fetch_public_snapshot returned Ok(PublicSnapshot) for a controlled backend whose leaderboard endpoint always served revision A and reports endpoint always served distinct revision B. The two read pairs were equal, so the function accepted an A+B snapshot despite no single published revision containing that pair.
   • Cause
     • Lines 73-81 compare only successive parsed composite snapshots (next == prev). This detects changes between attempts but has no shared revision/version/ETag or atomic endpoint to establish that leaderboard and reports belong to the same publication revision.
   • Fix
     • Require an atomic backend snapshot endpoint or a common revision token returned and validated on both route responses; reject or retry when the two route revision tokens differ. Repeated equality must not be treated as proof of cross-route revision consistency.

   _{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: 63fd5c3 | Re-trigger Greptile}

[cursor]

Rebased onto main after #202 squash-merged (GitHub had auto-closed this PR when the #202 head branch was deleted).

relearn-eval stays unpinned (eval_image_digest empty). Dead digests including sha256:cbc4bbb8 are not re-pinned.

Greptile P1 (torn /leaderboard + /reports snapshot): already fixed in this stack — fetch_public_snapshot re-reads until two consecutive parsed snapshots agree, or fails closed as BackendError::Inconsistent (fix(bounty): pin one feed revision before signing scores). Covered by a_feed_that_never_holds_still_is_never_signed_as_one_snapshot and a_feed_that_settles_is_scored_at_the_revision_it_settled_on.

[greptile-apps]

Stable torn snapshots pass the consecutive-composite check

Two equal reads only establish that each route stayed unchanged between attempts; they do not establish that /leaderboard and /reports came from the same backend publication. A backend that persistently exposes leaderboard revision A while reports remains on revision B produces the same A+B composite twice, so this returns it and signs results from a state the backend never published atomically. Require a shared revision token that is checked across both responses, or retrieve an atomic snapshot endpoint and reject mismatched revisions.

Artifacts

Stable torn snapshot reproduction harness source

• The captured temporary Rust integration harness starts a live backend with a revision-A leaderboard route and a distinct revision-B reports route, then calls fetch_public_snapshot; it defines the stable torn condition that must be rejected.

Stable torn snapshot harness command output

• The captured cargo test output shows the live route harness completed four requests and fetch_public_snapshot accepted the mixed A-plus-B snapshot; the P1 claim is reproduced.

_{Ran code and verified through T-Rex}

[cursor]

Fixed in 29da80b9. Consecutive equal pair-reads were not enough: a host that always served leaderboard revision A beside reports revision B would agree with itself.

fetch_public_snapshot now also requires /leaderboard valid_count to match the valid rows on /reports (snapshot_halves_agree). A stable torn pair is BackendError::Mismatched and burns. Covered by a_stable_torn_pair_is_never_signed_as_one_snapshot (live mock) and a_stable_torn_pair_does_not_agree.

No shared backend revision token is required; if both routes later grow an ETag, the moving-feed re-read can collapse to one round.

[cursor]

Follow-up is #206 (this PR already merged). Consecutive equals are no longer enough: valid_count must match the valid reports, and when both routes carry revision/snapshot_id/etag (or HTTP ETag) those tokens must be equal. Covered by a_stable_torn_pair_is_never_signed_as_one_snapshot and mismatched_publication_tokens_are_never_signed_as_one_snapshot.