Lock four-challenge fail-closed eval; unpin the dead relearn-eval image

[echobt]

Subnet 100 evaluation/incentive is fail-closed across the four live challenges. Emission and model IDs unchanged.

relearn-eval is unpinned on purpose (dd3b24c0)

sha256:cbc4bbb8… (f3cfa69) is dead. Do not re-pin or re-harvest it. The live harvest reached a pod and exited 1 without RELEARN_EVAL_OK: the CUDA scoring image shipped no vLLM, fell back to transformers, and then crashed because Qwen3VLVideoProcessor wants torchvision. No champion was recorded.

eval_image_digest and relearn_git_sha are now empty, which is the fail-closed state: POST /v1/submissions answers 503 eval image digest not pinned instead of renting a B200 that cannot score. No digest was invented to make the 503 go away.

The pin names every digest that reached a pod and failed, with why — cbc4bbb8…, 201cc5d2…, 303c6357…, 00839671…, 86240d76… — and pin_names_the_digests_that_must_not_be_re_harvested keeps that list from being quietly dropped. the_committed_pin_refuses_live_scoring_and_says_why calls scoring_readiness with the shipped pin, so the assertion is about host behaviour rather than a string in a TOML file.

Image and Agent pins are untouched (different images, both still green).

Gates that could be skipped by omitting evidence (7a048b71)

Three gates in judge_challenger were fail-open, so a run that shipped a thinner metrics document simply did not take them:

Gate	Was	Now
Perturbation	if let Some(..) on perturbed — no series, no gate	perturbation_evidence_missing
Base canaries	if let Some(..) on canaries — no series, no gate	base_canary_evidence_missing
Pixel shuffle	iterated the challenger's own map, so dropping a family skipped the control	shuffle_evidence_missing for any family the champion measured

The champion is now the reference for which vision families the holdout has images in, so "text-only holdout" still means no shuffle gate, but "challenger declined the control" does not.

BaselineMeasurement::verify additionally requires perturbed and canaries alongside public and general_canary. An image that never emits them is refused at boot, where the operator can act, instead of rejecting every challenger for a reason no miner can fix.

EvalError::Integrity is a 503, not a 500 — an unverified teardown is this host refusing to believe a run, not the miner's mistake. eval_err now matches EvalError exhaustively so a new failure mode has to be given a status deliberately.

Tests for the whole path (c02f9741, 8982b4fb)

a_live_run_takes_every_gate_or_scores_zero walks it in one run: champion recorded → can_score: true → a clean submission scored on the Lium backend with a non-zero lattice and a displacing paired test → then every gate zeroing its own run (contamination, undeclared manifest, public–holdout gap, general-bench canary, and the three that used to be skippable). Each rejection is asserted against its own gate name, so failing closed everywhere would not pass.

the_next_challenger_is_judged_against_the_promoted_champion covers the bar moving: a later challenger at the same skill ties the sitting champion instead of beating a base model it never faced.

no_single_gate_can_be_dodged_by_dropping_its_evidence in relearn-score blanks each series in turn and requires a zero lattice every time.

Product lock (do not regress)

• Live: relearn 4000 / relearn-image 1500 / relearn-agent 1500 / bounty 3000. relearn-mm off.
• Base Qwen/Qwen3.8-27B, teacher incoai/GLM-5.3-NVFP4 local-dir, no DFlash.
• Empty digest → 503, never sim. RELEARN_FORCE_SIM is CI/local only and is false in env-prod.yml / env-staging.yml.
• Holdout missing or mismatched → 503. can_score needs holdout loaded and champion recorded.
• Contamination rejected before the rent. No judge URL and no base weights both refuse before the rent. Harvest rents pin.eval_image@digest only, never a Prism recipe. test -x /usr/bin/relearn-eval probes before the holdout is staged.

Verification

cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, and xtask loc-cap / consensus-lint / spec-check / design-check / external-docs-check all pass. cargo deny was not run (binary absent on this VM); no dependency changed — the only new item on a public API is a pub use re-export of ExampleSeries from relearn-score.

Ops blockers (not code, not fixable here)

1. Need a new GHCR digest for ghcr.io/cortexlm/relearn-eval whose CUDA scoring layer ships vLLM and torchvision. That work is in CortexLM/relearn, not this repo. Until it lands, relearn submissions 503 and no champion can be recorded. Bump eval_image_digest and relearn_git_sha together, from a commit reachable on that repo's default branch.
2. Lium harvest — RELEARN_ALLOW_MODEL_DOWNLOAD=1 on the harvest host for the first champion pull (never in git). After Lium caches the weights, pin RELEARN_BASE_MODEL_DIR to that pod path and unset ALLOW.
3. Teacher URL on the host — RELEARN_TEACHER_API_URL is still required; without it the harvest refuses before renting.
4. Prod holdout must rotate off the CI seal (private salt and private catalog, then re-sign the trust root).

Do not merge from this agent. Do not retune emission or model IDs.