fix(challenges): fail-closed live holdout, report auth, relearn persist - #203
Draft
echobt wants to merge 59 commits into
Draft
fix(challenges): fail-closed live holdout, report auth, relearn persist#203echobt wants to merge 59 commits into
echobt wants to merge 59 commits into
Conversation
Retire Design and Prism as live products. Wire Relearn HTTP submit, displacement scoring, digest-freeze holdout, and operator promote. Keep Lium/receipt/paired-test rails. Pin CortexLM/relearn (seed in-tree until org write can create the public repo). Co-authored-by: Mathis <echobt@users.noreply.github.com>
Workspace clippy -D warnings failed on Relearn arena mapping. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Keep miners-pay-Lium as the trust line. Leave holdout-after-freeze in the eval loop internals, not README/miner Trust rows. Co-authored-by: Mathis <echobt@users.noreply.github.com>
README is what Cortex is plus two links. Validators get a one-screen role; Relearn miners get submit, Lium, promote-wait, and a pointer. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Both audience pages live under docs/external-miner/. README still has two links only. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Teacher URL, model, and key come from RELEARN_TEACHER_* only. Missing URL or key skips to sim. No baked host. Miners pay Lium. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Wire a second live challenge (id bounty) on the Relearn PR branch: pair a Bittensor hotkey to a dedicated Cortex Chat account, file bug reports, and score by precision vs the previous champion. Default emission is relearn 7000 / bounty 3000. Chat inject stays env-only; no teacher hosts or secrets in git. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Cortex reads CortexLM/backend GET /v1/bounty/public/* for scoring. It does not serve a public leaderboard. Unset URL skips (CI). Co-authored-by: Mathis <echobt@users.noreply.github.com>
Pin nvidia/Cosmos3-Super-Text2Image (OpenMDW 1.1, verified card) as the generator seed miners fine-tune, and Q-Judger (Qwen/Qwen-Image-Bench, Apache-2.0) as the only judge. Flux-family bases are refused outright. Eval prompts are frozen in config/relearn-t2i-pin.toml so no miner brings its own upsampler to the scored split, and every miner generates the same prompt ids at the same derived seeds. The holdout split is present in git only as a commitment; records come from an operator file and are verified at boot, so a wrong file refuses submissions instead of scoring the public split. Gates: paired displacement on the holdout, per-L1-pillar regression epsilon, seed replay, agentic faithfulness agreement, contamination, and a judge N/A ceiling. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Miners attach a permissively licensed vision encoder plus a projector to the champion Relearn LLM. Pin google/siglip2-so400m-patch14-384 (verified Apache-2.0) and restrict miner-supplied encoders to OSI-permissive terms. Two gates, both mandatory. Gate 1 reruns the existing Relearn text holdout on the submitted LM with vision ignored; a drop past epsilon zeroes the submission no matter how good the vision numbers are, and an encoder-only submission must additionally hash-match the champion LM. Gate 2 is a frozen image holdout (captioning, VQA, OCR, spatial relations - not ImageNet or COCO test) plus agentic image-tool traces, each replayed with the pixels shuffled so a model that ignores the image cannot pass. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Retune challenges.toml to relearn 4000 / relearn-t2i 1500 / relearn-mm 1500 / bounty 3000 with a dedicated public key per challenge, re-signed under a fresh throwaway owner key (the previous owner secret is off-git by design). Wire relearn-t2i-challenge on :8097 and relearn-mm-challenge on :8098 through the Dockerfile, compose matrix, env examples, GHCR image lanes, remote-deploy, and local-e2e. Local smoke materializes the T2I holdout from the documented dev salt; without it the service still answers /health and 503s submissions, which is the intended fail-closed state. Docs: README lists all four challenges, docs/RELEARN-T2I.md and docs/RELEARN-MM.md carry the control-plane contracts, and the miner guides state that Flux is rejected, Q-Judger is the only T2I judge, and encoder licenses must be OSI-permissive. external-docs-check now pins those claims. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The site matched emission shares by slug and only knew relearn, so after the four-way split the marketing surface would have shown one challenge's 4000 bps as the whole subnet. Add both arena slugs (wire form equals the trust-root challenge id), frames, and status fetches, and drop the retired design/prism arena list that both landing and metrics were still feeding. A down challenge backend leaves the static frame in place rather than dropping the arena, so the emission column still accounts for every challenge. Added a regression test that every live arena slug resolves to a non-zero trust-root share and that the shares sum to 10000. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Pin a holdout commitment on the live relearn path, load records only from RELEARN_HOLDOUT_FILE, and reject submissions when the file is missing or does not match. Add contamination, public-gap, vision pixel-shuffle, and off-lattice canary-regression gates. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Overlay locked Hugging Face ids on Testeur's holdout gates. Keep public_ids, holdout_commitment, and holdout_size. vLLM serves from RELEARN_TEACHER_LOCAL_DIR, never the repo id. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Template, .greptile rules, and a short AGENTS / miner-docs note. Covers fail-closed, holdout-off-git, no Modal/secrets, no Flux, OSI encoders, and BASE_* / domain-tag freeze. Additive; scoring gates unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Relock teacher_nvfp4 to incoai/GLM-5.3-NVFP4 (full GLM-5.3). Wire id glm-5.3, not flash. Serve from RELEARN_TEACHER_LOCAL_DIR. Keep #199 holdout_commitment, public_ids, and holdout_size. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live submit could not produce a real verdict. `eval_after_freeze` always
called `sim_slice_scores`, so a host with an empty `eval_image_digest`
answered `201 CREATED` with simulated numbers, and an empty miner manifest
skipped the contamination gate entirely.
Eval backend is now explicit, mirroring relearn-mm:
- `EvalBackend::{Lium, Sim}` + `force_sim()` / `resolve_eval_backend()`.
Sim is selected only by `RELEARN_FORCE_SIM`, never as a fallback.
- `eval_after_freeze` takes the pin and the backend. On `Lium` it returns
`EvalImageUnpinned` without a `sha256:` pin and `LiveHarvestUnavailable`
with one; both map to `503`, so no sim number can reach the lattice.
- `/v1/status` publishes `eval_backend`, `force_sim`, and `can_score`; the
submit row publishes `eval_backend`.
- The sim base champion is only seeded on a host that resolved `Sim`.
Contamination now carries its evidence:
- `ContaminationEvidence` records the declared id / image-hash / dataset
counts next to the hits, and `contamination_evidence` replaces
`contaminated_fingerprints`.
- An undeclared manifest is `GateFail::ContaminationEvidenceMissing`, the
same fail-closed shape as `PublicEvidenceMissing`.
The offline harness also could not express any promotion: perturbed and
holdout were drawn from independent salts, so the drop was ~0.17 against a
0.05 ceiling and every sim run failed `Perturbation`. Sim slices are now
derived from one skill level per artifact (`sim_artifact_skill`), with the
retention slices offset from the holdout draw, so `RELEARN_FORCE_SIM=1`
can reach `awaiting_admin` and the gate tests compare like with like.
Pin ids and `judge_challenger` thresholds are untouched.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
The pin and COMPLETENESS named the local dev salt next to the committed `holdout_commitment`, which reads as the production seal. Both now describe it as the CI / local commitment, point at `config/CEREMONY.md`, and keep the requirement that production rotate salt *and* catalog before re-signing. The salt literal stays only in `deploy/scripts/local-e2e.sh`, where it is the local default; `committed_pin.rs` asserts it is absent from the pin. Also documents the two fail-closed behaviours this branch adds: the 503 a host returns without a `sha256:` eval-image pin (with `eval_backend` / `can_score` on `/v1/status`), and that an undeclared miner `manifest` fails the contamination gate. Miner example manifest now declares real fields instead of three empty arrays, which would now be rejected. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Found on the live HTTP run: `--force-sim` bound `env = "RELEARN_FORCE_SIM"` to a clap `bool`, so the documented `RELEARN_FORCE_SIM=1` was rejected with `invalid value '1' for '--force-sim'` and the service exited at boot. Only compose's `"true"` / `"false"` strings worked, which is why it went unnoticed. Drops the env binding, matching `relearn-mm-challenge` and `relearn-t2i-challenge`: the value is read by `resolve_eval_backend`, which accepts `1` / `true` / `yes`. A test pins every documented spelling. Co-authored-by: Mathis <echobt@users.noreply.github.com>
`cargo test --workspace` is red on the parent branch: #200 moved `relearn_challenge_task::BASE_MODEL_ID` to `Qwen/Qwen3.8-27B`, but `relearn-mm-task` re-exports that constant as `LM_BASE_MODEL_ID` while `config/relearn-mm-pin.toml` and four tests still spelled out `Qwen/Qwen3.8-Flash-Next`. Eight tests failed, including `RelearnMmPin::validate` on the committed pin. The encoder attaches to the Relearn champion's LM, so the MM pin has to carry the live LLM base — this aligns it rather than choosing a new model. `lm_side_tracks_the_relearn_champion_base` now asserts against `relearn_challenge_task::BASE_MODEL_ID` so the two cannot drift again. Also updates the miner-facing seed harness default (`docs/external-miner/relearn-seed/eval/harness/eval.py`), which still pointed `RELEARN_BASE_MODEL` at Flash-Next. Drop this commit if #200 fixes the fallout upstream. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
…' into cursor/relearn-fail-closed-live-eval-61d5 Co-authored-by: Mathis <echobt@users.noreply.github.com>
Two holes the live replay of d7c5920 found behind the digest 503. **A live host could never record a champion baseline.** Making the sim baseline sim-only was right — a live challenger judged against simulated champion scores is not a comparison — but it left the live path with no baseline at all. The moment a `sha256:` eval image is pinned, submit answers `no champion baseline recorded` before contamination, public–holdout gap, or pixel-shuffle can run, so every gate this branch exists to protect is dead code on the only host that matters. Boot now records the baseline with the scorer the host will actually use: - `LiveScorer` is the eval-image harvest seam. `eval_after_freeze` takes it and refuses (`LiveHarvestUnavailable`) rather than substituting sim. - `BaselineMeasurement` (`RELEARN_BASE_CHAMPION_FILE`) is the operator's recorded measurement of the base model. `verify` binds it to the pin's `eval_image_digest` and `holdout_commitment` and refuses a measurement missing a series the gates read, so a junk baseline fails at boot instead of silently rejecting every challenger. - `boot_base_champion` picks the recorded measurement, else the wired harvest, else refuses. Sim numbers are not a candidate on a live host. **Refusals banked rows.** The row was inserted before the eval, so every 503 left an `evaluating` row with no scores that appears on no operator surface — spammable. Nothing is persisted until scoring produced a verdict, and the row is inserted once in its final state instead of insert-then-patch. Check order now puts the root cause first and spends nothing it cannot use: holdout unseal, then `scoring_readiness` (digest pin + harvest), then the baseline, then the eval. An unpinned digest still reports the pin, not the baseline, and a live eval never runs for a submission the host could not judge — that eval is the miner's Lium spend. `/v1/status` gains `live_harvest_wired` and `champion_baseline_recorded` next to `can_score`, since "cannot score" now has three distinct causes. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Adds the operator surface for the live baseline: what `RELEARN_BASE_CHAMPION_FILE` is, the JSON the eval image has to emit, and why `public` / `general_canary` cannot be empty (a champion the gates cannot read rejects every challenger for a reason the miner cannot act on). Also records the two new `/v1/status` fields (`live_harvest_wired`, `champion_baseline_recorded`) and the three distinct causes of `can_score: false` in the miner troubleshoot table, notes that a refusal persists no row, and adds the operator checklist item plus COMPLETENESS gaps for the unwired live harvest. Co-authored-by: Mathis <echobt@users.noreply.github.com>
`pick_honors_compute_override` sets a process-wide env var that `pick_compute` reads, so it raced `pick_b200_then_fallback`: whichever ran while the override was set picked `L40S` and failed. It flaked in two of three `cargo test --workspace` runs and passed in isolation, which made it hard to tell whether an unrelated PR was green. Test-only mutex around both. Pre-existing and unrelated to this branch — drop this commit if it lands elsewhere. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
`live_scorer` was hardcoded `None`, so a digest-pinned host still refused
with `live_harvest_wired: false` and the champion could only come from
`RELEARN_BASE_CHAMPION_FILE`. This adds the control-plane client.
The scoring code is not here: it ships inside `eval_image` from
CortexLM/relearn. `relearn-lium-harvest` is the client for it.
- `LiveScorer` is now async (`#[async_trait]`, as `EvalJobBackend` already
is), so `eval_after_freeze` and `boot_base_champion` are too. The bin
builds its runtime before recording the baseline.
- `RelearnEvalMetrics` is the image contract: the `BaselineMeasurement`
envelope plus run identity, so the operator's baseline file is literally
the image's output for the base model — one format, not two. `verify`
binds schema, `submission_digest`, and `artifact_digest`, so a pod cannot
answer with another artifact's numbers or replay an earlier run.
- `LiumHarvest` owns the lifecycle: boot the digest-pinned image, deliver
the request, harvest, then terminate and require verified teardown before
any score is accepted. An orphan pod outranks the run result because it
keeps spending the miner's money.
- `LiumEvalPod` is the Lium/SSH transport. Run inputs travel in
`request.json` over stdin, never interpolated into the remote command, so
a crafted digest cannot be shell injection on the pod. The workdir is
scrubbed after the run; termination is the real guarantee.
- `build_live_scorer` wires it on `EvalBackend::Lium` only, from
`LIUM_API_KEY`. Sim scores in-process and never gets a pod. No key means
no harvest and the host refuses.
Nothing in this crate can compute a score. A pod that returns no
`RELEARN_EVAL_OK`, no metrics document, or a document bound to another run
is an error, not a fallback — the tests feed metrics-document fixtures
through the real parse/verify path rather than substituting sim numbers.
`prism-lium` additionally re-exports `ssh_exec{,_allow_fail,_stdin}` and
`SshExecOutput` so the client can reuse its retry/timeout helpers.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
…ract Live replay of the wiring found the spec left `ssh_public_keys` empty, so `provision` refused with "Lium rent requires at least one SSH public key" — the harvest could never boot a pod. A pod with no master key is also unreachable, so the request could not be delivered and no metrics could be read back. - `LiumHarvest` takes the master public key(s) and registers them under `relearn-eval-worker`; it refuses before renting when there is none. - The bin loads it from `LIUM_SSH_PUBLIC_KEY_FILE` (same convention as `prism-challenge`) and does not wire the harvest without it. - `LIUM_API_BASE_URL` can point a staging host at a stand-in provider instead of spending real money. - Provider and eval-image failures are now 503, not 500: the host cannot score right now, which is not the miner's mistake, and retrying is right. Documents the image contract in `docs/RELEARN.md` — entrypoint, marker lines, document shape, and the identity/pin checks — so `CortexLM/relearn` has something to implement against. Also records, in RELEARN.md, COMPLETENESS.md, and the operator checklist, that the harvest request carries the holdout to a miner-paid pod: mitigated by the digest-pinned image, tmp delivery, post-run scrub, and verified termination, but not eliminated. That exposure is an owner decision, not an implementation detail. Co-authored-by: Mathis <echobt@users.noreply.github.com>
CortexLM/relearn PR #2 published the image that implements the harvest contract, so the eval half of live scoring is now pinned: eval_image_digest = sha256:303c6357… (was empty) relearn_git_sha = 82e21442… (was 6e952d1e) Only those two values change. base_model, teacher_model, teacher_nvfp4, teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size, and public_ids are byte-for-byte unchanged. `committed_pin_cannot_rent_or_score_live_yet` existed to make this moment deliberate; it is replaced by `committed_pin_allows_live_rent` plus `eval_image_is_digest_only`, which pins the exact digest and asserts the tag never migrates into `eval_image`. Pinning the image is not the whole job, and the docs now say so: a live host still needs the harvest wired and a champion baseline recorded, each with its own 503 and boot-log line. `remote-deploy.sh` warns when a master host is missing the relearn holdout or the baseline, since with the image pinned those two files are the difference between scoring and 503 on every submission. Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned SHA is not yet reachable from that repo's default branch. Re-pin to the merge commit when it lands. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The boot warning read "submissions will 503 unless RELEARN_BASE_CHAMPION_FILE covers the baseline and a harvest is configured", which is wrong now that the digest is pinned: the replay showed a host with the baseline recorded and no harvest still 503ing every submission. The baseline covers the champion only; each submission needs its own measurement. Points at the env example instead. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Digest-only pins from CortexLM/relearn PR #3, harvest contracts that speak those images (shared RELEARN_METRICS markers, recorded traces, optional image canary), and miner/operator docs for the four live ids. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Keeps the function under clippy's 100-line cap after the both-empty skip, without weakening the one-sided fail-closed rule. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Primary eval_image is ghcr.io/cortexlm/relearn-image-eval at the relearn#3 digest; t2i-eval stays the same-digest alias. Teacher ids on relearn are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Match #201 HEAD: sha256:00839671… / 822d2729. Refuse sha256:303c6357… which never printed the marker. Image, agent, and bounty pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Status was advertising can_score on sim/live hosts with no verified episodes while submit already 503d. Holdout is now the first readiness check. LLM digest and image/bounty pins are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Port the #201 LLM harvest gate onto the harvest-pod transport: without RELEARN_TEACHER_API_URL, can_score is false and submit 503s naming the var, with no Lium rent. Teacher env is delivered over stdin as teacher.env (umask 077, set -a) rather than interpolated into the remote command. Missing-marker 503s include a redacted run.log tail. Image/agent harvests stay env-free. LLM digest, image/agent pins, and bounty fail-closed are unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Move the scoring_readiness teacher-gate fixture out of the test body so items-after-statements stays clean. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Image and Agent harvest now refuse without a judge/teacher URL, forward stdin teacher.env the same way LLM does, and reject contaminated or empty-evidence manifests before renting. Bounty treats any prior fingerprint as duplicate and refuses title==body / token-thin reports. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Pre-eval reject helpers satisfy too-many-arguments/field-reassign lints. Image happy-path submits a declared manifest so it still evals instead of being screened as empty evidence. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live status advertised can_score while submit 503'd without a champion baseline. Harvest run_cmd now prefers /usr/bin/<eval> (else command -v) so a login-less SSH PATH cannot miss the binary. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Replace sha256:00839671 (exit 127 on a login-less SSH PATH) with sha256:86240d76 from CortexLM/relearn 9998154f. Image/agent pins unchanged. Do not pin sha256:303c6357. Co-authored-by: Mathis <echobt@users.noreply.github.com>
sha256:201cc5d2 ships /usr/bin/relearn-eval (relearn d107a7c1). Do not pin 86240d76, 00839671, or 303c6357. Image/agent pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Harvest was setting image_digest and then renting prism-recipe-v10, so SSH ran /usr/bin/relearn-eval on the wrong image (exit 127). InstanceSpec now carries docker_image + startup; provision uses that pin and refuses a recipes fallback. Probe the score binary before staging the holdout. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The live relearn-eval image already CMD ["serve"] (tini + entrypoint). Create the Lium template with docker_image=pin@digest, no tag, no startup, and a 12-hex name so recipes cannot be reused. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Lock the LLM eval image to sha256:cbc4bbb8 (git f3cfa69). Reject the four earlier judge-broken prefixes. LLM and Image can_score now require the holdout file to be loaded, matching Agent. Bounty HTTP rejects a second miner refiling a closed fingerprint. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Reword the GLM-5.3 pin comment so the committed-pin secret scan does not treat max_tokens as a leaked credential. Add HTTP tests that a commitment-mismatched holdout cannot load, so status stays can_score false and submit is 503 on LLM, Image, and Agent. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The eval image does not bake Qwen. TeacherEnv now forwards RELEARN_BASE_MODEL_DIR, HF_HOME, HF_HUB_CACHE, and an explicit RELEARN_ALLOW_MODEL_DOWNLOAD=1 into teacher.env over stdin. ready() refuses to rent when neither DIR nor ALLOW is set. Status publishes base_weights.primed + via (var name only). ALLOW_DOWNLOAD is never defaulted. Pin stays cbc4bbb8. Co-authored-by: Mathis <echobt@users.noreply.github.com>
A teacher/judge URL alone is no longer enough to score. Binary readiness tests now fail closed without DIR/ALLOW, then pass with the first-champion ALLOW_DOWNLOAD=1 path. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Lium NCU hosts reject split even when min_gpu_count_for_rental is 1. Parse ncu_profiling_enabled and rent the whole host instead of posting gpu_count=1 on the live 2× B200. A 1-GPU harvest pin still matches. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Refuse the public CI fixture as a live holdout pin, require operator auth on bounty report reads, and persist Relearn MemoryStore across restarts with fail-closed corrupt restore. Co-authored-by: Mathis <echobt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Security + rebase of the four-challenge stack (#198–#202) onto
main. GitHub #202 isCONFLICTINGbecause its base iscursor/relearn-fail-closed-live-eval-61d5, notmain.origin/main(803859fa) is already an ancestor of the stack head (d516c679), so retargeting tomainis clean — fail-closed eval behavior is unchanged.This PR then fixes every live security P1 still present in that tree:
Live holdouts are public fixtures (Greptile P1 on
config/relearn-agent-pin.toml, same class on LLM + Image). The committed pin stays a labeled CI/local fixture. Live (non-sim) scoring refuses if the effective commitment equals that fixture. Operators must supply a private commitment via env / secret-store file (RELEARN_*_HOLDOUT_COMMITMENT/_FILE). Missing override → warn,can_score=false, submit 503. Override equal to the fixture is refused. No private catalogue, salt, holdout set, or “secret” commitment is committed.Unauthenticated bounty report GETs (
GET /v1/reportsandGET /v1/reports/{id}). Operator bearer required (same as adjudicate). Empty hashes → 503auth_unconfigured. Missing/wrong bearer → 401 and no report body / repro / account / hotkey. Gateway defense in depth: GET report reads 403 (POST submit stays open). Tests assert unauthenticated GET is 401.Relearn
MemoryStorelost on restart.RELEARN_STATE_FILEpersists submissions, scores, champion, and base champion. Missing file = first boot. Present + corrupt = fail closed (StoreError::Restore), not empty-score. Restart + corrupt-restore regressions added. Holdout records are not persisted.Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo fmt --all -- --check-D warningson the changed cratesxtask loc-cap,spec-check,design-check,external-docs-check,consensus-lintcargo test --workspace(CI green on1610f624)cargo deny check(CI green on1610f624)Risk
/challenge/{id}/v1/reports*now 403s; miners still POST/v1/reports.RELEARN_STATE_FILEon the host volume or it starts empty again after restart.BASE_*rename, no emission-weight retune, no crypto domain change.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags, unless this PR’s purpose is a coordinatedcutover documented in
docs/NAMING.md.