feat(relearn): pin Qwen3.8-27B + incoai/GLM-5.3-NVFP4 on #199 - #200
Draft
echobt wants to merge 3 commits into
Draft
feat(relearn): pin Qwen3.8-27B + incoai/GLM-5.3-NVFP4 on #199#200echobt wants to merge 3 commits into
echobt wants to merge 3 commits into
Conversation
Overlay locked Hugging Face ids on Testeur's holdout gates. Keep public_ids, holdout_commitment, and holdout_size. vLLM serves from RELEARN_TEACHER_LOCAL_DIR, never the repo id. Co-authored-by: Mathis <echobt@users.noreply.github.com>
cursor
Bot
force-pushed
the
cursor/relearn-anti-overfit-ec6c
branch
from
August 31, 2026 08:28
f6c6514 to
dbaf686
Compare
cursor
Bot
changed the title
cursor
Bot
changed the base branch from
cursor/relearn-subnet-v0-fbf6
to
cursor/relearn-anti-overfit-gates-893c
August 31, 2026 08:28
Template, .greptile rules, and a short AGENTS / miner-docs note. Covers fail-closed, holdout-off-git, no Modal/secrets, no Flux, OSI encoders, and BASE_* / domain-tag freeze. Additive; scoring gates unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Relock teacher_nvfp4 to incoai/GLM-5.3-NVFP4 (full GLM-5.3). Wire id glm-5.3, not flash. Serve from RELEARN_TEACHER_LOCAL_DIR. Keep #199 holdout_commitment, public_ids, and holdout_size. Co-authored-by: Mathis <echobt@users.noreply.github.com>
cursor
Bot
changed the title
8 tasks
cursor Bot
pushed a commit
that referenced
this pull request
Aug 31, 2026
`cargo test --workspace` is red on the parent branch: #200 moved `relearn_challenge_task::BASE_MODEL_ID` to `Qwen/Qwen3.8-27B`, but `relearn-mm-task` re-exports that constant as `LM_BASE_MODEL_ID` while `config/relearn-mm-pin.toml` and four tests still spelled out `Qwen/Qwen3.8-Flash-Next`. Eight tests failed, including `RelearnMmPin::validate` on the committed pin. The encoder attaches to the Relearn champion's LM, so the MM pin has to carry the live LLM base — this aligns it rather than choosing a new model. `lm_side_tracks_the_relearn_champion_base` now asserts against `relearn_challenge_task::BASE_MODEL_ID` so the two cannot drift again. Also updates the miner-facing seed harness default (`docs/external-miner/relearn-seed/eval/harness/eval.py`), which still pointed `RELEARN_BASE_MODEL` at Flash-Next. Drop this commit if #200 fixes the fallout upstream. Co-authored-by: Mathis <echobt@users.noreply.github.com>
echobt
added a commit
that referenced
this pull request
Sep 3, 2026
…ge (#202) * feat(relearn): one-challenge subnet v0 eval loop Retire Design and Prism as live products. Wire Relearn HTTP submit, displacement scoring, digest-freeze holdout, and operator promote. Keep Lium/receipt/paired-test rails. Pin CortexLM/relearn (seed in-tree until org write can create the public repo). Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(site): satisfy clippy assigning-clones and match-same-arms Workspace clippy -D warnings failed on Relearn arena mapping. Co-authored-by: Mathis <echobt@users.noreply.github.com> * chore(relearn): pin CortexLM/relearn git sha * docs(relearn): drop anti-TEE slogans from user-facing copy Keep miners-pay-Lium as the trust line. Leave holdout-after-freeze in the eval loop internals, not README/miner Trust rows. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): split miner and validator one-pagers README is what Cortex is plus two links. Validators get a one-screen role; Relearn miners get submit, Lium, promote-wait, and a pointer. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): keep miner and validator pages together Both audience pages live under docs/external-miner/. README still has two links only. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): env-only teacher http api Teacher URL, model, and key come from RELEARN_TEACHER_* only. Missing URL or key skips to sim. No baked host. Miners pay Lium. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): add bounty challenge beside relearn Wire a second live challenge (id bounty) on the Relearn PR branch: pair a Bittensor hotkey to a dedicated Cortex Chat account, file bug reports, and score by precision vs the previous champion. Default emission is relearn 7000 / bounty 3000. Chat inject stays env-only; no teacher hosts or secrets in git. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): consume backend public api Cortex reads CortexLM/backend GET /v1/bounty/public/* for scoring. It does not serve a public leaderboard. Unset URL skips (CI). Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-t2i): add Relearn T2I challenge on Cosmos3 + Q-Judger Pin nvidia/Cosmos3-Super-Text2Image (OpenMDW 1.1, verified card) as the generator seed miners fine-tune, and Q-Judger (Qwen/Qwen-Image-Bench, Apache-2.0) as the only judge. Flux-family bases are refused outright. Eval prompts are frozen in config/relearn-t2i-pin.toml so no miner brings its own upsampler to the scored split, and every miner generates the same prompt ids at the same derived seeds. The holdout split is present in git only as a commitment; records come from an operator file and are verified at boot, so a wrong file refuses submissions instead of scoring the public split. Gates: paired displacement on the holdout, per-L1-pillar regression epsilon, seed replay, agentic faithfulness agreement, contamination, and a judge N/A ceiling. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-mm): add Relearn Multimodal challenge beside Relearn LLM Miners attach a permissively licensed vision encoder plus a projector to the champion Relearn LLM. Pin google/siglip2-so400m-patch14-384 (verified Apache-2.0) and restrict miner-supplied encoders to OSI-permissive terms. Two gates, both mandatory. Gate 1 reruns the existing Relearn text holdout on the submitted LM with vision ignored; a drop past epsilon zeroes the submission no matter how good the vision numbers are, and an encoder-only submission must additionally hash-match the champion LM. Gate 2 is a frozen image holdout (captioning, VQA, OCR, spatial relations - not ImageNet or COCO test) plus agentic image-tool traces, each replayed with the pixels shuffled so a model that ignores the image cannot pass. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(challenges): split emission four ways and wire the new services Retune challenges.toml to relearn 4000 / relearn-t2i 1500 / relearn-mm 1500 / bounty 3000 with a dedicated public key per challenge, re-signed under a fresh throwaway owner key (the previous owner secret is off-git by design). Wire relearn-t2i-challenge on :8097 and relearn-mm-challenge on :8098 through the Dockerfile, compose matrix, env examples, GHCR image lanes, remote-deploy, and local-e2e. Local smoke materializes the T2I holdout from the documented dev salt; without it the service still answers /health and 503s submissions, which is the intended fail-closed state. Docs: README lists all four challenges, docs/RELEARN-T2I.md and docs/RELEARN-MM.md carry the control-plane contracts, and the miner guides state that Flux is rejected, Q-Judger is the only T2I judge, and encoder licenses must be OSI-permissive. external-docs-check now pins those claims. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(site): list Relearn T2I and Relearn Multimodal as arenas The site matched emission shares by slug and only knew relearn, so after the four-way split the marketing surface would have shown one challenge's 4000 bps as the whole subnet. Add both arena slugs (wire form equals the trust-root challenge id), frames, and status fetches, and drop the retired design/prism arena list that both landing and metrics were still feeding. A down challenge backend leaves the static frame in place rather than dropping the arena, so the emission column still accounts for every challenge. Added a regression test that every live arena slug resolves to a non-zero trust-root share and that the shares sum to 10000. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): fail-closed holdout and anti-overfit gates Pin a holdout commitment on the live relearn path, load records only from RELEARN_HOLDOUT_FILE, and reject submissions when the file is missing or does not match. Add contamination, public-gap, vision pixel-shuffle, and off-lattice canary-regression gates. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): pin 27b and glm flash nvfp4 on #199 Overlay locked Hugging Face ids on Testeur's holdout gates. Keep public_ids, holdout_commitment, and holdout_size. vLLM serves from RELEARN_TEACHER_LOCAL_DIR, never the repo id. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(repo): complete greptile review gate on the overlay Template, .greptile rules, and a short AGENTS / miner-docs note. Covers fail-closed, holdout-off-git, no Modal/secrets, no Flux, OSI encoders, and BASE_* / domain-tag freeze. Additive; scoring gates unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): pin teacher to incoai glm-5.3 nvfp4 Relock teacher_nvfp4 to incoai/GLM-5.3-NVFP4 (full GLM-5.3). Wire id glm-5.3, not flash. Serve from RELEARN_TEACHER_LOCAL_DIR. Keep #199 holdout_commitment, public_ids, and holdout_size. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): fail closed instead of scoring submissions in sim Live submit could not produce a real verdict. `eval_after_freeze` always called `sim_slice_scores`, so a host with an empty `eval_image_digest` answered `201 CREATED` with simulated numbers, and an empty miner manifest skipped the contamination gate entirely. Eval backend is now explicit, mirroring relearn-mm: - `EvalBackend::{Lium, Sim}` + `force_sim()` / `resolve_eval_backend()`. Sim is selected only by `RELEARN_FORCE_SIM`, never as a fallback. - `eval_after_freeze` takes the pin and the backend. On `Lium` it returns `EvalImageUnpinned` without a `sha256:` pin and `LiveHarvestUnavailable` with one; both map to `503`, so no sim number can reach the lattice. - `/v1/status` publishes `eval_backend`, `force_sim`, and `can_score`; the submit row publishes `eval_backend`. - The sim base champion is only seeded on a host that resolved `Sim`. Contamination now carries its evidence: - `ContaminationEvidence` records the declared id / image-hash / dataset counts next to the hits, and `contamination_evidence` replaces `contaminated_fingerprints`. - An undeclared manifest is `GateFail::ContaminationEvidenceMissing`, the same fail-closed shape as `PublicEvidenceMissing`. The offline harness also could not express any promotion: perturbed and holdout were drawn from independent salts, so the drop was ~0.17 against a 0.05 ceiling and every sim run failed `Perturbation`. Sim slices are now derived from one skill level per artifact (`sim_artifact_skill`), with the retention slices offset from the holdout draw, so `RELEARN_FORCE_SIM=1` can reach `awaiting_admin` and the gate tests compare like with like. Pin ids and `judge_challenger` thresholds are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): say the committed holdout commitment is not the live seal The pin and COMPLETENESS named the local dev salt next to the committed `holdout_commitment`, which reads as the production seal. Both now describe it as the CI / local commitment, point at `config/CEREMONY.md`, and keep the requirement that production rotate salt *and* catalog before re-signing. The salt literal stays only in `deploy/scripts/local-e2e.sh`, where it is the local default; `committed_pin.rs` asserts it is absent from the pin. Also documents the two fail-closed behaviours this branch adds: the 503 a host returns without a `sha256:` eval-image pin (with `eval_backend` / `can_score` on `/v1/status`), and that an undeclared miner `manifest` fails the contamination gate. Miner example manifest now declares real fields instead of three empty arrays, which would now be rejected. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): let RELEARN_FORCE_SIM=1 boot instead of failing to parse Found on the live HTTP run: `--force-sim` bound `env = "RELEARN_FORCE_SIM"` to a clap `bool`, so the documented `RELEARN_FORCE_SIM=1` was rejected with `invalid value '1' for '--force-sim'` and the service exited at boot. Only compose's `"true"` / `"false"` strings worked, which is why it went unnoticed. Drops the env binding, matching `relearn-mm-challenge` and `relearn-t2i-challenge`: the value is read by `resolve_eval_backend`, which accepts `1` / `true` / `yes`. A test pins every documented spelling. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-mm): follow the locked LLM base id instead of Flash-Next `cargo test --workspace` is red on the parent branch: #200 moved `relearn_challenge_task::BASE_MODEL_ID` to `Qwen/Qwen3.8-27B`, but `relearn-mm-task` re-exports that constant as `LM_BASE_MODEL_ID` while `config/relearn-mm-pin.toml` and four tests still spelled out `Qwen/Qwen3.8-Flash-Next`. Eight tests failed, including `RelearnMmPin::validate` on the committed pin. The encoder attaches to the Relearn champion's LM, so the MM pin has to carry the live LLM base — this aligns it rather than choosing a new model. `lm_side_tracks_the_relearn_champion_base` now asserts against `relearn_challenge_task::BASE_MODEL_ID` so the two cannot drift again. Also updates the miner-facing seed harness default (`docs/external-miner/relearn-seed/eval/harness/eval.py`), which still pointed `RELEARN_BASE_MODEL` at Flash-Next. Drop this commit if #200 fixes the fallout upstream. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): tell seed miners to declare their training manifest Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): record a live champion baseline and stop banking refusals Two holes the live replay of d7c5920 found behind the digest 503. **A live host could never record a champion baseline.** Making the sim baseline sim-only was right — a live challenger judged against simulated champion scores is not a comparison — but it left the live path with no baseline at all. The moment a `sha256:` eval image is pinned, submit answers `no champion baseline recorded` before contamination, public–holdout gap, or pixel-shuffle can run, so every gate this branch exists to protect is dead code on the only host that matters. Boot now records the baseline with the scorer the host will actually use: - `LiveScorer` is the eval-image harvest seam. `eval_after_freeze` takes it and refuses (`LiveHarvestUnavailable`) rather than substituting sim. - `BaselineMeasurement` (`RELEARN_BASE_CHAMPION_FILE`) is the operator's recorded measurement of the base model. `verify` binds it to the pin's `eval_image_digest` and `holdout_commitment` and refuses a measurement missing a series the gates read, so a junk baseline fails at boot instead of silently rejecting every challenger. - `boot_base_champion` picks the recorded measurement, else the wired harvest, else refuses. Sim numbers are not a candidate on a live host. **Refusals banked rows.** The row was inserted before the eval, so every 503 left an `evaluating` row with no scores that appears on no operator surface — spammable. Nothing is persisted until scoring produced a verdict, and the row is inserted once in its final state instead of insert-then-patch. Check order now puts the root cause first and spends nothing it cannot use: holdout unseal, then `scoring_readiness` (digest pin + harvest), then the baseline, then the eval. An unpinned digest still reports the pin, not the baseline, and a live eval never runs for a submission the host could not judge — that eval is the miner's Lium spend. `/v1/status` gains `live_harvest_wired` and `champion_baseline_recorded` next to `can_score`, since "cannot score" now has three distinct causes. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): document the champion baseline the live gates need Adds the operator surface for the live baseline: what `RELEARN_BASE_CHAMPION_FILE` is, the JSON the eval image has to emit, and why `public` / `general_canary` cannot be empty (a champion the gates cannot read rejects every challenger for a reason the miner cannot act on). Also records the two new `/v1/status` fields (`live_harvest_wired`, `champion_baseline_recorded`) and the three distinct causes of `can_score: false` in the miner troubleshoot table, notes that a refusal persists no row, and adds the operator checklist item plus COMPLETENESS gaps for the unwired live harvest. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(prism-verda): serialize the two PRISM_VERDA_COMPUTE tests `pick_honors_compute_override` sets a process-wide env var that `pick_compute` reads, so it raced `pick_b200_then_fallback`: whichever ran while the override was set picked `L40S` and failed. It flaked in two of three `cargo test --workspace` runs and passed in isolation, which made it hard to tell whether an unrelated PR was green. Test-only mutex around both. Pre-existing and unrelated to this branch — drop this commit if it lands elsewhere. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(prism-verda): use the method reference clippy asks for Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): wire the live harvest on the Lium path `live_scorer` was hardcoded `None`, so a digest-pinned host still refused with `live_harvest_wired: false` and the champion could only come from `RELEARN_BASE_CHAMPION_FILE`. This adds the control-plane client. The scoring code is not here: it ships inside `eval_image` from CortexLM/relearn. `relearn-lium-harvest` is the client for it. - `LiveScorer` is now async (`#[async_trait]`, as `EvalJobBackend` already is), so `eval_after_freeze` and `boot_base_champion` are too. The bin builds its runtime before recording the baseline. - `RelearnEvalMetrics` is the image contract: the `BaselineMeasurement` envelope plus run identity, so the operator's baseline file is literally the image's output for the base model — one format, not two. `verify` binds schema, `submission_digest`, and `artifact_digest`, so a pod cannot answer with another artifact's numbers or replay an earlier run. - `LiumHarvest` owns the lifecycle: boot the digest-pinned image, deliver the request, harvest, then terminate and require verified teardown before any score is accepted. An orphan pod outranks the run result because it keeps spending the miner's money. - `LiumEvalPod` is the Lium/SSH transport. Run inputs travel in `request.json` over stdin, never interpolated into the remote command, so a crafted digest cannot be shell injection on the pod. The workdir is scrubbed after the run; termination is the real guarantee. - `build_live_scorer` wires it on `EvalBackend::Lium` only, from `LIUM_API_KEY`. Sim scores in-process and never gets a pod. No key means no harvest and the host refuses. Nothing in this crate can compute a score. A pod that returns no `RELEARN_EVAL_OK`, no metrics document, or a document bound to another run is an error, not a fallback — the tests feed metrics-document fixtures through the real parse/verify path rather than substituting sim numbers. `prism-lium` additionally re-exports `ssh_exec{,_allow_fail,_stdin}` and `SshExecOutput` so the client can reuse its retry/timeout helpers. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): require the master SSH key and document the image contract Live replay of the wiring found the spec left `ssh_public_keys` empty, so `provision` refused with "Lium rent requires at least one SSH public key" — the harvest could never boot a pod. A pod with no master key is also unreachable, so the request could not be delivered and no metrics could be read back. - `LiumHarvest` takes the master public key(s) and registers them under `relearn-eval-worker`; it refuses before renting when there is none. - The bin loads it from `LIUM_SSH_PUBLIC_KEY_FILE` (same convention as `prism-challenge`) and does not wire the harvest without it. - `LIUM_API_BASE_URL` can point a staging host at a stand-in provider instead of spending real money. - Provider and eval-image failures are now 503, not 500: the host cannot score right now, which is not the miner's mistake, and retrying is right. Documents the image contract in `docs/RELEARN.md` — entrypoint, marker lines, document shape, and the identity/pin checks — so `CortexLM/relearn` has something to implement against. Also records, in RELEARN.md, COMPLETENESS.md, and the operator checklist, that the harvest request carries the holdout to a miner-paid pod: mitigated by the digest-pinned image, tmp delivery, post-run scrub, and verified termination, but not eliminated. That exposure is an owner decision, not an implementation detail. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy(relearn): pin the published eval image digest and relearn SHA CortexLM/relearn PR #2 published the image that implements the harvest contract, so the eval half of live scoring is now pinned: eval_image_digest = sha256:303c6357… (was empty) relearn_git_sha = 82e21442… (was 6e952d1e) Only those two values change. base_model, teacher_model, teacher_nvfp4, teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size, and public_ids are byte-for-byte unchanged. `committed_pin_cannot_rent_or_score_live_yet` existed to make this moment deliberate; it is replaced by `committed_pin_allows_live_rent` plus `eval_image_is_digest_only`, which pins the exact digest and asserts the tag never migrates into `eval_image`. Pinning the image is not the whole job, and the docs now say so: a live host still needs the harvest wired and a champion baseline recorded, each with its own 503 and boot-log line. `remote-deploy.sh` warns when a master host is missing the relearn holdout or the baseline, since with the image pinned those two files are the difference between scoring and 503 on every submission. Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned SHA is not yet reachable from that repo's default branch. Re-pin to the merge commit when it lands. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): stop implying the baseline file can replace the harvest The boot warning read "submissions will 503 unless RELEARN_BASE_CHAMPION_FILE covers the baseline and a harvest is configured", which is wrong now that the digest is pinned: the replay showed a host with the baseline recorded and no harvest still 503ing every submission. The baseline covers the champion only; each submission needs its own measurement. Points at the env example instead. Co-authored-by: Mathis <echobt@users.noreply.github.com> * refactor(harvest): share one digest-pinned lium pod transport Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-image): fail-closed live eval on the relearn-image id Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-agent): add the replayed-tool-trace challenge Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-agent): pin the episode set and add the ceremony helper Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): price bugs by severity and fail closed on ingest Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(challenges): relock four live ids and re-sign the trust root Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy(challenges): bring up the four live services and gate mm off Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(challenges): name the four live challenges and their gates Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): pin image/agent evals to relearn#3 Digest-only pins from CortexLM/relearn PR #3, harvest contracts that speak those images (shared RELEARN_METRICS markers, recorded traces, optional image canary), and miner/operator docs for the four live ids. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-image): split canary gating out of judge_challenger Keeps the function under clippy's 100-line cap after the both-empty skip, without weakening the one-sided fail-closed rule. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-image): name the published image-eval pin Primary eval_image is ghcr.io/cortexlm/relearn-image-eval at the relearn#3 digest; t2i-eval stays the same-digest alias. Teacher ids on relearn are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin the eval image that prints RELEARN_EVAL_OK Match #201 HEAD: sha256:00839671… / 822d2729. Refuse sha256:303c6357… which never printed the marker. Image, agent, and bounty pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-agent): can_score stays false until holdout loads Status was advertising can_score on sim/live hosts with no verified episodes while submit already 503d. Holdout is now the first readiness check. LLM digest and image/bounty pins are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): forward teacher env and fail before rent Port the #201 LLM harvest gate onto the harvest-pod transport: without RELEARN_TEACHER_API_URL, can_score is false and submit 503s naming the var, with no Lium rent. Teacher env is delivered over stdin as teacher.env (umask 077, set -a) rather than interpolated into the remote command. Missing-marker 503s include a redacted run.log tail. Image/agent harvests stay env-free. LLM digest, image/agent pins, and bounty fail-closed are unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): hoist Unready scorer stub for clippy Move the scoring_readiness teacher-gate fixture out of the test body so items-after-statements stays clean. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): fail-before-rent and screen junk before eval Image and Agent harvest now refuse without a judge/teacher URL, forward stdin teacher.env the same way LLM does, and reject contaminated or empty-evidence manifests before renting. Bounty treats any prior fingerprint as duplicate and refuses title==body / token-thin reports. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): clippy persist helpers and image happy-path Pre-eval reject helpers satisfy too-many-arguments/field-reassign lints. Image happy-path submits a declared manifest so it still evals instead of being screened as empty evidence. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): can_score waits for champion; resolve eval PATH Live status advertised can_score while submit 503'd without a champion baseline. Harvest run_cmd now prefers /usr/bin/<eval> (else command -v) so a login-less SSH PATH cannot miss the binary. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin eval image that ships /usr/bin/relearn-eval Replace sha256:00839671 (exit 127 on a login-less SSH PATH) with sha256:86240d76 from CortexLM/relearn 9998154f. Image/agent pins unchanged. Do not pin sha256:303c6357. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin the CUDA scoring image CI pulled sha256:201cc5d2 ships /usr/bin/relearn-eval (relearn d107a7c1). Do not pin 86240d76, 00839671, or 303c6357. Image/agent pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): rent digest-pinned harvest templates not recipes Harvest was setting image_digest and then renting prism-recipe-v10, so SSH ran /usr/bin/relearn-eval on the wrong image (exit 127). InstanceSpec now carries docker_image + startup; provision uses that pin and refuses a recipes fallback. Probe the score binary before staging the holdout. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): harvest template is repo@digest with no startup The live relearn-eval image already CMD ["serve"] (tini + entrypoint). Create the Lium template with docker_image=pin@digest, no tag, no startup, and a 12-hex name so recipes cannot be reused. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin GLM-5.3 judge image; can_score needs holdout Lock the LLM eval image to sha256:cbc4bbb8 (git f3cfa69). Reject the four earlier judge-broken prefixes. LLM and Image can_score now require the holdout file to be loaded, matching Agent. Bounty HTTP rejects a second miner refiling a closed fingerprint. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): drop pin _token wording; HTTP mismatch 503 Reword the GLM-5.3 pin comment so the committed-pin secret scan does not treat max_tokens as a leaked credential. Add HTTP tests that a commitment-mismatched holdout cannot load, so status stays can_score false and submit is 503 on LLM, Image, and Agent. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): forward base weights; fail closed before rent The eval image does not bake Qwen. TeacherEnv now forwards RELEARN_BASE_MODEL_DIR, HF_HOME, HF_HUB_CACHE, and an explicit RELEARN_ALLOW_MODEL_DOWNLOAD=1 into teacher.env over stdin. ready() refuses to rent when neither DIR nor ALLOW is set. Status publishes base_weights.primed + via (var name only). ALLOW_DOWNLOAD is never defaulted. Pin stays cbc4bbb8. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): bin ready() tests require ALLOW_DOWNLOAD A teacher/judge URL alone is no longer enough to score. Binary readiness tests now fail closed without DIR/ALLOW, then pass with the first-champion ALLOW_DOWNLOAD=1 path. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): rent whole-host ncu b200 for 1-gpu pin Lium NCU hosts reject split even when min_gpu_count_for_rental is 1. Parse ncu_profiling_enabled and rent the whole host instead of posting gpu_count=1 on the live 2× B200. A 1-GPU harvest pin still matches. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): unpin cbc4bbb8; no eval image has scored cbc4bbb8 (f3cfa69) exited 1 on a rented pod without RELEARN_EVAL_OK: no vLLM on the CUDA scoring image, transformers fallback, then Qwen3VLVideoProcessor crashed for want of torchvision. An empty digest is the fail-closed state - submissions 503 rather than renting a B200 that cannot score. The pin now names every digest that reached a pod and failed, and the tests assert the fail-closed state instead of a digest. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): fail closed when gate evidence is missing Perturbation and the known-answer canaries were 'if let Some(..)': a run whose eval document omitted the series took neither gate. Pixel shuffle iterated the challenger's own map, so dropping a family skipped the control. All three now fail closed, with the champion as the reference for which vision families the holdout actually has images in. BaselineMeasurement::verify additionally requires perturbed + canaries, so an image that never emits them is refused at boot rather than rejecting every challenger for a reason the miner cannot act on. EvalError::Integrity is a 503, not a 500: an unverified teardown is this host refusing to believe a run, not the miner's mistake. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): unpinned eval image and fail-closed gate evidence RELEARN.md, COMPLETENESS.md, and the miner-facing external-miner docs now say the digest is empty on purpose and what the next image needs (vLLM + torchvision), and list the new *_evidence_missing rejections so a miner can tell a skipped gate from a failed one. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(relearn): cover the live path end to end through HTTP One run walks champion recorded -> can_score true -> a clean submission scored on the Lium backend with lattice -> every gate zeroing its own run: contamination, undeclared manifest, public-holdout gap, general canary, and the three that used to be skippable by omitting the series. Each rejection is asserted against its own gate name, so failing closed everywhere would not pass. Second test: promoting moves the bar. A later challenger at the same skill ties the sitting champion instead of beating the base model. relearn-score re-exports ExampleSeries so a consumer can build the slices whose types are already in its public API. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): split judge gates for clippy; name every eval status judge_challenger crossed the 100-line cap once the missing-evidence arms landed, so the retention and pixel-shuffle gates are their own functions. eval_err now matches EvalError exhaustively: a new failure mode has to be given a status deliberately instead of inheriting a 500. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(relearn): committed pin refuses live scoring at runtime Asserting the digest string is empty says the file is right; calling scoring_readiness with the shipped pin says the host is. The reported cause has to be the pin, not a downstream symptom. Co-authored-by: Mathis <echobt@users.noreply.github.com> * chore(ci): retrigger checks after retargeting to main Co-authored-by: Mathis <echobt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mathis <echobt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on Testeur’s PR #199. This PR does not rewrite the holdout / contamination / shuffle / canary gates.
Pin overlay (live)
base_modelQwen/Qwen3.8-27B(unchanged)teacher_nvfp4incoai/GLM-5.3-NVFP4— full GLM-5.3 NVFP4 (~433 GiB, 97 files, license other). NotLibertAIDAI/GLM-5.3-Flash-NVFP4. Never DFlash2 (CC BY-NC-ND).teacher_modelglm-5.3(notglm-5.3-flash)RELEARN_TEACHER_LOCAL_DIR— never pass the Hugging Face repo id to vLLMrelearnHugging Face card checked 2026-08-31:
incoai/GLM-5.3-NVFP4HTTP 200, ungated, 97 siblings,base_model: zai-org/GLM-5.3.Kept from #199:
public_ids,holdout_commitment(84aed154…),holdout_size = 120.crates/relearn-scoreandholdout.rsunchanged.Greptile (already on this branch)
Template checkbox +
.greptile/+ short AGENTS / miner-docs note.Test plan
relearn-challenge-task,relearn-evalcommitted_pin,relearn-http)xtask external-docs-check@greptileai reviewif silent)