fix(relearn): fail closed instead of scoring submissions in sim

[echobt]

Summary

Follow-up on #200 (which sits on #199). Makes the Relearn submit path fail closed instead of scoring in sim, and gets a live host all the way to the gates. No model id or lock is touched — teacher_nvfp4 = "incoai/GLM-5.3-NVFP4" (wire id glm-5.3), base_model = "Qwen/Qwen3.8-27B", teacher_backend, and #199's holdout_commitment / holdout_size / public_ids are byte-for-byte unchanged. The only pin values this branch moves are eval_image_digest and relearn_git_sha. T2I, Bounty, and encoder-MM stay off. No secrets, teacher hostnames, or Modal anywhere.

Round 1 — sim was the only scorer

eval_after_freeze unconditionally called sim_slice_scores, and the pin shipped eval_image_digest = "", so a prod-shaped host answered 201 CREATED with simulated numbers. The backend is now explicit (EvalBackend::{Lium, Sim}), sim is reachable only through RELEARN_FORCE_SIM, and /v1/status plus every submit row publish which scorer ran.

contaminated_fingerprints returned [] for an empty manifest, which judge_challenger read as a clean run. ContaminationEvidence now carries the declared counts next to the hits, and an undeclared manifest is GateFail::ContaminationEvidenceMissing.

Docs no longer present the local salt as the live seal.

Round 2 — the gates were unreachable

A live host could never record a champion baseline. Baseline seeding was sim-only, so pinning a sha256: image made every submit answer no champion baseline recorded before contamination, gap, or shuffle could run. Boot now records the baseline with the scorer the host uses: the operator-recorded BaselineMeasurement (RELEARN_BASE_CHAMPION_FILE, verified against the pin's eval_image_digest and holdout_commitment), else the wired harvest. Sim numbers are not a candidate on a live host.

Refusals banked rows. Nothing is persisted until scoring produced a verdict; the row is inserted once in its final state. Check order is holdout → scoring_readiness → baseline → eval → persist, so an unpinned digest reports the pin rather than a downstream symptom, and a live eval never runs for a submission the host could not judge.

Round 3 — the harvest is wired

live_scorer was hardcoded None. New crate crates/relearn-lium-harvest is the control-plane client; the scoring code stays in CortexLM/relearn.

• LiveScorer is async (#[async_trait], as EvalJobBackend already is), so eval_after_freeze and boot_base_champion are too.
• RelearnEvalMetrics is the image contract: the BaselineMeasurement envelope plus run identity, so the operator's baseline file is literally the image's output for the base model — one format, not two. verify binds schema, submission_digest, and artifact_digest, so a pod cannot answer with another artifact's numbers or replay an earlier run.
• LiumHarvest owns the lifecycle: boot eval_image@<digest> with the master SSH key under price/GPU/lifetime guardrails, deliver request.json over stdin, read RELEARN_METRICS= + RELEARN_EVAL_OK, scrub, terminate, and require verified teardown before accepting any score — an orphan pod keeps spending the miner's money, so it outranks the run result.
• LiumEvalPod is the Lium/SSH transport. Run inputs never touch the remote command line, so a crafted digest cannot be shell injection on the pod.
• build_live_scorer wires it on EvalBackend::Lium only, from LIUM_API_KEY + LIUM_SSH_PUBLIC_KEY_FILE. Sim scores in-process and never gets a pod; a missing key means no harvest and the host refuses.
• LIUM_API_BASE_URL lets staging point at a stand-in provider.
• Provider / image failures are 503, not 500.

Nothing in the crate can compute a score. No RELEARN_EVAL_OK, no document, or a document bound to another run is an error — the tests push metrics-document fixtures through the real parse/verify/lifecycle path rather than substituting sim numbers.

The image contract (entrypoint, markers, document shape, identity checks) is documented in docs/RELEARN.md § Eval image contract.

Round 4 — first published digest (superseded)

CortexLM/relearn PR #2 first published sha256:303c63573c… (82e21442…). That image booted on Lium but printed no RELEARN_EVAL_OK. Do not pin it. See Round 6.

committed_pin_cannot_rent_or_score_live_yet existed to make the first pin deliberate. It is replaced by committed_pin_allows_live_rent (pins the exact digest) and eval_image_is_digest_only (asserts the tag never migrates into eval_image).

Pinning the image is not the whole job, and the docs now say so: a live host still needs the harvest wired and a champion baseline recorded, each with its own 503. remote-deploy.sh warns when a master host is missing the relearn holdout or the baseline, since those two files are now the difference between scoring and 503 on every submission.

Round 5 — the pod could not see the teacher

Live run: harvest wired, pod RUNNING, no RELEARN_EVAL_OK. Confirmed root cause — a Lium InstanceSpec has no env field, so provisioning cannot set pod environment and run_cmd exported nothing. The pod inherited nothing from the master, the image found no RELEARN_TEACHER_API_URL, and it exited without scoring.

Env now reaches the image the only way it can, over the SSH exec:

• TeacherEnv::from_host_env reads RELEARN_TEACHER_API_URL, RELEARN_TEACHER_MODEL, RELEARN_TEACHER_API_KEY off the live host. Names only in git; values are operator state.
• They travel in teacher.env over stdin (umask 077) and are sourced with set -a before the entrypoint — not interpolated into the remote command, where the key would sit in the pod's process table on hardware the miner pays for.
• RELEARN_TEACHER_MODEL defaults to the pin's teacher_model and RELEARN_BASE_MODEL comes from the pin, so the image and the pin cannot disagree.

Missing config now fails before the rent. LiveScorer grows a ready() hook that scoring_readiness consults, so a host with no judge reports can_score: false and 503s naming RELEARN_TEACHER_API_URL instead of paying for a pod that cannot score.

When the marker is missing anyway, the image's log comes back. The remote command echoes exit=$rc and the last 8 KiB of run.log; the control plane returns that tail in the 503 (2 KiB) and logs it (8 KiB), with the metrics document and marker lines stripped. The tail is redacted against the forwarded secrets first — the pod was handed the teacher key and this body is miner-visible.

Round 6 — repin to the image that prints RELEARN_EVAL_OK

The first published digest (sha256:303c63573c…) is the one that printed no marker. Only these two fields move; eval_image stays ghcr.io/cortexlm/relearn-eval. Do not pin sha256:303c63573c….

Field	Was (round 4)	Now
eval_image_digest	sha256:303c63573c9492b1214fa8071977b26a96acc25eaba258c2958c4773a3cd5e45	sha256:0083967170ddc0fc892857ef8df12db93ff40e3c62b30ded6459d26df5c9498a
relearn_git_sha	82e2144266aee3ffdd276396905820bc4e53f3b5	822d272972ef92c0a92ef2e2a9ba51a54f5165fd

committed_pin_allows_live_rent now asserts the new digest and the new git SHA, and refuses the old digest substring. Model ids, teacher, holdout commitment, public ids, and eval_image are unchanged.

Owner decision to make

RELEARN_TEACHER_API_KEY crosses to the same pod. The image judges over HTTP and the pod inherits nothing, so if the teacher API requires auth the credential has to be forwarded. It goes over stdin, is scrubbed after the run, and is redacted out of anything surfaced — but a miner-controlled pod could still spend the operator's teacher quota. Scope and rate-limit that credential, or leave it unset when the pod can reach the teacher without auth (nothing is forwarded then). Recorded in RELEARN.md § Pod environment, COMPLETENESS.md, and the operator checklist.

The harvest request carries the holdout items to a miner-paid pod. A model cannot be scored on prompts it is not shown, so live scoring on rented hardware means the private split is on that pod for the run. Mitigations: digest-pinned image, delivery into /tmp/relearn_eval rather than a persisted path, post-run scrub, verified termination. That narrows the window; it does not close it — the commitment makes drift detectable, not exfiltration. Recorded in RELEARN.md, COMPLETENESS.md, and the operator checklist. If that exposure is unacceptable, keep the challenge on the recorded-baseline path and treat live scoring as gated on an in-enclave design.

Also found

• RELEARN_FORCE_SIM=1 crashed the service at boot — the env var was bound to a clap bool, which rejects 1. Binding dropped, matching the sibling bins.
• The offline harness could not express any promotion — perturbed and holdout were drawn from independent salts (~0.17 drop against a 0.05 ceiling), so submit_eval_promote_happy_path never entered its promote block. Sim slices now derive from one skill level per artifact; judge_challenger and every gate constant untouched.
• cargo test --workspace was already red on feat(relearn): pin Qwen3.8-27B + incoai/GLM-5.3-NVFP4 on #199 #200 — relearn-mm-task re-exports BASE_MODEL_ID while its pin and four tests still said Flash-Next. Aligned; the test now asserts against the constant.
• prism-verda flake — pick_honors_compute_override raced pick_b200_then_fallback over a process-wide env var. Test-only mutex.

The last two are unrelated to this branch and are in their own commits — drop them if they land elsewhere.

Greptile

Every PR is reviewed by Greptile before merge. Config: .greptile/.

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Test plan

• cargo test -p relearn-eval --test committed_pin after this repin
• cargo test --workspace --no-fail-fast — clean (pre-repin)
• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• xtask loc-cap / consensus-lint / spec-check / design-check / external-docs-check
• Live HTTP replay against the previous digest (no digest override), harvest pointed at a stand-in provider (LIUM_API_BASE_URL=http://127.0.0.1:1) so nothing was rented for real

Live case	Result
Committed pin, no harvest	/v1/status serves the real digest + relearn_git_sha; champion_baseline_recorded: true from the operator file, live_harvest_wired: false, can_score: false; submit → 503 … no in-process sim; 0 rows
Committed pin + harvest wired	can_score: true; submit reaches the harvest — 503 backend: lium transport … from the stand-in provider, not a sim score; 0 rows
Baseline measured by a different image	refused at boot: measured by eval image "sha256:abab…"; champion_baseline_recorded: false
RELEARN_FORCE_SIM=1 on the pinned image	eval_backend: sim, no pod; declared manifest → awaiting_admin, eligible: true
No digest / no key / no SSH pubkey	harvest not wired, can_score: false, submit 503 with the root cause; 0 rows banked
No RELEARN_TEACHER_API_URL (reproduces the live failure)	boot warns; can_score: false with live_harvest_wired: true; submit → 503 backend: RELEARN_TEACHER_API_URL not set …; no pod rented
Teacher configured	boot logs the forwarded variable names; can_score: true; submit proceeds to the rent
Leak check	teacher URL and key appear in neither /v1/status nor the boot log

Harvest tests (relearn-lium-harvest): the_teacher_env_is_forwarded_into_the_pod, no_judge_url_refuses_before_renting_a_pod, a_missing_marker_surfaces_the_redacted_log_tail, an_empty_log_still_says_something_useful, redaction_needs_a_credential_sized_secret, the_log_tail_drops_the_metrics_document, env_names_are_reported_without_values, the_pod_env_is_sourced_from_a_file_not_the_command_line, a_non_zero_exit_is_reported_in_stdout, boots_the_pinned_digest_and_returns_the_image_numbers, refuses_without_a_digest_pin_and_never_boots, refuses_without_a_master_ssh_key_and_never_boots, always_tears_the_pod_down_even_when_the_run_fails, unverified_teardown_is_an_integrity_failure, a_document_for_another_run_is_refused, a_run_without_the_ok_marker_is_refused, silence_from_the_pod_is_never_a_score, extract_accepts_the_marker_line_or_a_bare_document, the_run_command_harvests_the_sidecar_and_the_marker, the_request_never_reaches_the_shell, the_workdir_is_scrubbed_after_the_run, run_timeout_has_a_floor.

Wiring tests (relearn-challenge-bin): live_harvest_is_wired_on_the_lium_path_not_on_sim, a_wired_harvest_makes_a_pinned_host_ready_to_score, live_boot_records_the_champion_baseline_from_the_operator_file, live_boot_without_a_baseline_source_records_nothing, sim_boot_still_records_the_sim_baseline, documented_force_sim_values_do_not_break_argument_parsing.

Pin tests (relearn-eval): committed_pin_allows_live_rent (exact new digest + SHA; refuses 303c63573c), eval_image_is_digest_only, committed_pin_has_the_locked_model_ids, pin_carries_no_endpoint_or_secret_or_t2i_salt, pin_says_its_commitment_is_not_the_live_seal.

Earlier rounds' tests are unchanged and still pass (fail-closed 503s, contamination evidence, no banked rows, sim opt-in).

Risk

Behavioural, and deliberately. The previous 201 on a prod-shaped host was a sim verdict; it is now a 503 until the host can really score. With the image pinned, the remaining blockers are operator state (harvest + baseline + teacher), each with its own 503 and boot-log line. Local and e2e compose set RELEARN_FORCE_SIM=true and keep working.

Operator requirements on a live host: RELEARN_BASE_CHAMPION_FILE (or a reachable harvest) for the baseline, LIUM_API_KEY and LIUM_SSH_PUBLIC_KEY_FILE for the harvest, and RELEARN_TEACHER_API_URL for the image's judge. All are visible on /v1/status (champion_baseline_recorded, live_harvest_wired, can_score) and in the boot log with the reason. None is committed.

Deploy continuity: relearn_git_sha is the head of CortexLM/relearn PR #2, which is open, not merged, so the pinned SHA is not reachable from that repo's default branch yet. Re-pin to the merge commit when it lands. Recorded in COMPLETENESS.md.

Not verified here: that the pinned digest exists in GHCR, and that the env file actually lands in the image's environment on a real pod. The env-file contents, the set -a sourcing order, the redaction, and the tail extraction are unit-tested; the live replay covers readiness, status, the 503 bodies, and the leak check. A real rent is still the only end-to-end proof.

prism-lium gains re-exports only (ssh_exec{,_allow_fail,_stdin}, SshExecOutput) — no behaviour change. No signature domain, emission share, or leaf format change; judge_challenger thresholds and SCORING_VERSION unchanged; relearn-challenge-task's holdout/commitment code untouched.

Recommendation, not implemented here: the contamination gate reads only the miner's declared manifest and the holdout, so it could be screened before the eval and save a contaminated submission a full pod rent — the same pre-pod screen Prism already does. I left it out because short-circuiting means constructing a PromoteVerdict outside judge_challenger, and a second verdict-construction path in a consensus-adjacent crate is not something to add as a drive-by cost optimisation.

Naming

I did not rename BASE_* environment variables, deployed host paths
(/opt/base, /run/base, …), GHCR baseintelligence/base package names, or
base-*-v1 cryptographic domain tags, unless this PR’s purpose is a coordinated
cutover documented in docs/NAMING.md.