Harness overhaul, phase A: correctness fixes, green CI, lean docs, evidence archive #68

Merged

Grok merged 6 commits from harness-overhaul into master 2026-09-25 10:09:41 +02:00

Conversation 2 Commits 6 Files changed 489 +3640 -36376

Grok commented 2026-09-25 09:43:59 +02:00

Owner

Copy link

Phase A of the harness overhaul. Owner directive (2026-09-25): targets are 1 and 2 cards, and nothing is "cannot" unless it is proven on hardware. This PR changes the repo only. It does not touch cfx-llm2, tenstorrent-serving.service or the cards; deploying is the owner's decision. Do not merge without review.

Commits

Step	Commit	What changed and why
A0	7e942d0	Correctness fixes: patch 0019 re-applied, the attention/tp.py mount fixed, and a sweep baseline that matches production (see below).
A1	987e88d	CI green (see below).
A2	93c9f73	Deletions and archiving (see below).
A3	e7bfc6d	Docs cut to the lean set; new LEDGER, STATUS and AGENTS.
A4	1aaaee7	bench/runs archive, INDEX.tsv, and a CI check that cited records exist.
—	347a3cb	Points the 1-card items at the new issue #69.

A1 detail:

• tt-metal is pinned to public upstream 60053f1f; the old pin was a local git+file URL.
• The Nix ttnn/sfpi/tt-metal packages moved to the host-local sub-flake nix/sim. The Nix ttnn is cp314 and cannot reach the cp310 image, and nixos-llm consumes only the modules. The ttnn store path is unchanged.
• Dead inputs are dropped.
• The formatter and shellcheck are pinned through --inputs-from ..

File counts, 682dd38 → HEAD:

• docs 136 → 11
• patches 68 → 28
• nix 19 → 13
• docker 13 → 6
• scripts 20 → 17
• bench 574 → 601 (bench/runs: 184 live + 225 archived; the growth is the round2-logs move plus INDEX/MOVED)

A0: how the fixes were verified

Patch 0019.

• I diffed 583d1c1 against 89f75ce first. The 89f75ce rewrite of prefill_forward dropped _slots_prefilled_since_decode.
• The fix re-applies the patch verbatim with patch -p1.
• modules/tenstorrent-serving.nix now fails the readiness probe if QWEN36_PREFILLED_SLOTS_FIX engaged is missing while prefilledSlotsFix is on. This is a minimal hunk; the option names are unchanged.

The attention mount. scripts/vllm-tt-serve.sh prod profiles now mount exactly the unit's seven files, including attention-tp.py → attention/tp.py. Before this, SDPA streaming was silently off for every launcher-booted arm. I dry-ran it with a fake sudo, which confirmed:

• all seven mounts;
• an EXTRA_ENV overlay takes precedence;
• a missing source exits 2.

Banner check. The new scripts/verify-arm-banners.sh fails closed on any missing <FLAG> engaged banner. With a fake docker, a passing server returns rc 0 and a server missing the 0019 banner returns rc 3 (the check also stops it).

Sweep baseline. bench/ab/sweep-32k.sh baseline is now the prod-argmax profile, which is the unit's env and file set. The candidate is baseline + CAND_ENV. The sweep refuses to run while the service is active, and it never uses docker rm -f.

No hardware run was done. The re-applied tree has no new speed number.

Deletions and why

• Docker images vllm-tt, -overlay, -k34m, -k5, -k5rf and their verifiers: these are the K3-K5/M levers, and op-count reduction was measured worthless for TP=2 traced decode.
• Dead patches tt-metal/0001, 0002, 0009, 0010, 0013: they apply to nothing (not HEAD, not the pin, not the prod mount).
  • Also removed: the duplicate pr53587 diff and tt-inference-server/0001, since upstream has P300 natively.
  • The NVFP4 loader (0009/0010) is noted in #69 for recovery.
• The block-keyed prefix series vllm-tt-plugin/0002-0006, patches/vllm, prefix-tests: prefix caching shipped the vLLM-native way.
• Patches folded into prod-local-*: 0012, 0014-0018, 0020 and plugin 0001. patches/README.md maps each patch to the file that absorbed it.
  • 0021-0023 stay until mtp-thinkingcap and thinkingcap-baseline rebase.
• Other removals:
  • generated/, bench/wip, bench/matrix, lever*-ab.sh, prefill-sweep.sh, mem-*.sh, m4-tp2-generate.sh, run_tp_contract.py, root BLACKHOLE_GUIDE.md;
  • autoresearch.sh with its measure script and test. The old ledger itself called the DSpark autoresearch loop off-objective.
• Nix packages: tt-cli, the otel packages, tt-model-manager, vscode-toolkit, probes and the production-serving-recipe (it described production wrongly).
• 93 docs, including DIRECTIVES-LEDGER, DOC-STATUS, advisory/ and n3/. Everything still true is folded into the 11 new docs. Recover any of them with git show 682dd38:docs/<name>.
• Moved, not deleted:
  • docs/advisory/round2-logs → bench/runs/archive/round2-logs-20260912;
  • bench/fixures → bench/fixtures/dspark;
  • closed-work drivers → bench/archive/;
  • 182 uncited records → bench/runs/archive/ (see MOVED.tsv).
  • 8 zero-byte .jsonl files were deleted.
• Evidence check. Every deletion was grepped against master and against the ACTIVE branch diffs first. scripts/check-cited-records.py (new CI step) passes. The keep set also includes records cited by ACTIVE branches and by the #53/#55 campaign files.

Reopened closures

The rule: a closure needs an on-hardware record at its stated scope. docs/LEDGER.md now holds the Closed-claims table (class, scope, record, reopen condition) and a Reopened list. Each item below has a Forgejo comment:

Issue	What is reopened
#62	MLP bf4 and attn/GDN bf4 (the G2 result is the record)
#66	The DSpark device port (PR #23 is refuted)
#56	"≤25 ms unreachable" and "30 tok/s unreachable" (MTP K3 measured 33.0 ms)
#51	The untried even grids
#61	The EXL3 no-go, which is an estimate
#63	TP=1 MTP
#46	Told that the 0019 regression is fixed here

Also corrected: "one tt-metal process per host" is now one per card (two-card-coexistence-20260925T0625Z), and card locks and slots live only in the tt-card-slot skill.

Branches and PRs

• Archived: 31 DEAD remote branches were tagged archive/<name> and then deleted. Each was re-verified first: 27 are ancestors of master, and 4 squash-merged branches differ from master in 0 files.
• Re-homed: the coordinator worktree's uncommitted #53/#55 files are now on the new branch decode-campaign-53-55 (based on c76460b). The files are byte-identical and the worktree was not touched.
• Closed:
  • #23, superseded by #66;
  • #16, branch merged; Ornith is now a 1-card candidate;
  • #15, content merged; the open goals moved to the new #69 (1-card target: G4.3 context ladder, G4.4 accuracy, Step D bf4 coverage, and the reopened TP=1 items).
• No ACTIVE branch was deleted. The SUPERSEDED branches (goal-4-nvfp4-single-card, ttsim-scout, dspark-acceptance-rate, thatch/lane2-serve-gotchas) were left in place.

Migration notes for the active branches

• mtp-thinkingcap, thinkingcap-baseline: rebase and drop the 28 pre-squash serving-throughput commits, e.g. git rebase --onto origin/master <last duplicate>.
  • Otherwise patches/tt-metal/0021-0023 and the retired docs come back as modify/delete conflicts.
  • Take the deletions. Your ledger entry 0aaaaaaaaaaaaaaaaaa is in docs/LEDGER.md as D-2026-09-24-thinkingcap.
  • Your AGENTS.md edits are superseded by the new AGENTS.md.
• Branches that edit flake.nix or flake.lock (agy/issue-49, -50, -64, tt-quant, bfp-quant): take master's inputs.
  • issue-50 should drop its tt-metal narHash hunk and use nix develop ./nix/sim#runtime.
  • nodes.tt-metal.locked.rev still exists; it is now 60053f1f.
• bfpq and other arms that call vllm-tt-serve.sh prod-argmax with their own attention-tp.py via EXTRA_ENV: the overlay is honoured, but VERIFY=1 (the default) now blocks until the server is ready and refuses a missing banner. Pass VERIFY=0 and call verify-arm-banners.sh yourself.
• Branches that edit modules/tenstorrent-serving.nix: CI excludes it from nixfmt for now. Run nixfmt once in Phase B, after issue-45/49/50/64 land.
• agy/issue-36: it edits the retired UPSTREAM-WATCH-PLUGIN-METAL.md. Fold its plugin-pin audit into docs/BUILD.md.
• agy/issue-38: it adds docs/LIFT-AUDIT-TT-METAL-56119.md. Fold that into docs/BUILD.md or REFERENCES.md, not a new doc.
• repo-skills: perf-issue-advisor references docs/advisory/round2-logs, which is now bench/runs/archive/round2-logs-20260912.
• decode-campaign-53-55: rebase onto master after this PR. Its DIRECTIVES-LEDGER edit becomes a modify/delete conflict; take the deletion.

Left for Phase B/C

Phase B:

• Land the active branches.
• Consolidate the skills.
• Rewrite the CLAUDE.md and retired-doc references in the frozen paths: comments in bench/ab/*, modules/tenstorrent-serving.nix, patches/prod-local-metal, and patches/k|tt-metal.
• nixfmt the serving module.
• Archive the frozen-path drivers.
• Refresh PROD-SERVING-SOURCES.md after issue-64.

Phase C:

• ttx arm specs and per-card slots.
• Turn k2-tracy into a build arg.
• A tt-metal mirror to replace the numbered patches.

Owner decisions needed

1. Deploy the 0019 + mount fixes. nixos-llm master still pins tt-stack f4c3cda (09-15).
2. Enable the prefix arm in the unit (#48). The module default is off. SERVING.md notes that the unit's enablePrefixCaching block-size/pool setup has no record of its own.
3. The MTP identity-gap tolerance (≤0.875 logit) and the batch-width tolerance.
4. The ThinkingCap licence (PolyForm Small Business) for commercial use.
5. The #53 B=1 ≤25 ms gate versus makespan ranking.
6. Whether to keep tt-model-publish's unpushed local commit.

🤖 Generated with Claude Code

Phase A of the harness overhaul. Owner directive (2026-09-25): targets are 1 and 2 cards, and nothing is "cannot" unless it is proven on hardware. **This PR changes the repo only.** It does not touch cfx-llm2, `tenstorrent-serving.service` or the cards; deploying is the owner's decision. **Do not merge without review.** ## Commits | Step | Commit | What changed and why | |---|---|---| | A0 | 7e942d0 | Correctness fixes: patch 0019 re-applied, the attention/tp.py mount fixed, and a sweep baseline that matches production (see below). | | A1 | 987e88d | CI green (see below). | | A2 | 93c9f73 | Deletions and archiving (see below). | | A3 | e7bfc6d | Docs cut to the lean set; new LEDGER, STATUS and AGENTS. | | A4 | 1aaaee7 | bench/runs archive, `INDEX.tsv`, and a CI check that cited records exist. | | — | 347a3cb | Points the 1-card items at the new issue #69. | **A1 detail:** - tt-metal is pinned to public upstream `60053f1f`; the old pin was a local `git+file` URL. - The Nix ttnn/sfpi/tt-metal packages moved to the host-local sub-flake `nix/sim`. The Nix ttnn is cp314 and cannot reach the cp310 image, and nixos-llm consumes only the modules. The ttnn store path is unchanged. - Dead inputs are dropped. - The formatter and shellcheck are pinned through `--inputs-from .`. **File counts, 682dd38 → HEAD:** - docs 136 → 11 - patches 68 → 28 - nix 19 → 13 - docker 13 → 6 - scripts 20 → 17 - bench 574 → 601 (bench/runs: 184 live + 225 archived; the growth is the `round2-logs` move plus INDEX/MOVED) ## A0: how the fixes were verified **Patch 0019.** - I diffed 583d1c1 against 89f75ce first. The 89f75ce rewrite of `prefill_forward` dropped `_slots_prefilled_since_decode`. - The fix re-applies the patch verbatim with `patch -p1`. - `modules/tenstorrent-serving.nix` now fails the readiness probe if `QWEN36_PREFILLED_SLOTS_FIX engaged` is missing while `prefilledSlotsFix` is on. This is a minimal hunk; the option names are unchanged. **The attention mount.** `scripts/vllm-tt-serve.sh` prod profiles now mount exactly the unit's seven files, including `attention-tp.py → attention/tp.py`. Before this, SDPA streaming was silently off for every launcher-booted arm. I dry-ran it with a fake `sudo`, which confirmed: - all seven mounts; - an EXTRA_ENV overlay takes precedence; - a missing source exits 2. **Banner check.** The new `scripts/verify-arm-banners.sh` fails closed on any missing `<FLAG> engaged` banner. With a fake docker, a passing server returns rc 0 and a server missing the 0019 banner returns rc 3 (the check also stops it). **Sweep baseline.** `bench/ab/sweep-32k.sh` baseline is now the `prod-argmax` profile, which is the unit's env and file set. The candidate is baseline + `CAND_ENV`. The sweep refuses to run while the service is active, and it never uses `docker rm -f`. **No hardware run was done.** The re-applied tree has no new speed number. ## Deletions and why - **Docker images** `vllm-tt`, `-overlay`, `-k34m`, `-k5`, `-k5rf` and their verifiers: these are the K3-K5/M levers, and op-count reduction was measured worthless for TP=2 traced decode. - **Dead patches** `tt-metal/0001`, `0002`, `0009`, `0010`, `0013`: they apply to nothing (not HEAD, not the pin, not the prod mount). - Also removed: the duplicate `pr53587` diff and `tt-inference-server/0001`, since upstream has P300 natively. - The NVFP4 loader (0009/0010) is noted in #69 for recovery. - **The block-keyed prefix series** `vllm-tt-plugin/0002-0006`, `patches/vllm`, `prefix-tests`: prefix caching shipped the vLLM-native way. - **Patches folded into prod-local-***: `0012`, `0014-0018`, `0020` and plugin `0001`. `patches/README.md` maps each patch to the file that absorbed it. - `0021-0023` stay until mtp-thinkingcap and thinkingcap-baseline rebase. - **Other removals:** - `generated/`, `bench/wip`, `bench/matrix`, `lever*-ab.sh`, `prefill-sweep.sh`, `mem-*.sh`, `m4-tp2-generate.sh`, `run_tp_contract.py`, root `BLACKHOLE_GUIDE.md`; - `autoresearch.sh` with its measure script and test. The old ledger itself called the DSpark autoresearch loop off-objective. - **Nix packages:** tt-cli, the otel packages, tt-model-manager, vscode-toolkit, probes and the production-serving-recipe (it described production wrongly). - **93 docs**, including DIRECTIVES-LEDGER, DOC-STATUS, `advisory/` and `n3/`. Everything still true is folded into the 11 new docs. Recover any of them with `git show 682dd38:docs/<name>`. - **Moved, not deleted:** - `docs/advisory/round2-logs` → `bench/runs/archive/round2-logs-20260912`; - `bench/fixures` → `bench/fixtures/dspark`; - closed-work drivers → `bench/archive/`; - 182 uncited records → `bench/runs/archive/` (see `MOVED.tsv`). - 8 zero-byte `.jsonl` files were deleted. - **Evidence check.** Every deletion was grepped against master and against the ACTIVE branch diffs first. `scripts/check-cited-records.py` (new CI step) passes. The keep set also includes records cited by ACTIVE branches and by the #53/#55 campaign files. ## Reopened closures The rule: **a closure needs an on-hardware record at its stated scope.** `docs/LEDGER.md` now holds the Closed-claims table (class, scope, record, reopen condition) and a Reopened list. Each item below has a Forgejo comment: | Issue | What is reopened | |---|---| | #62 | MLP bf4 and attn/GDN bf4 (the G2 result is the record) | | #66 | The DSpark device port (PR #23 is refuted) | | #56 | "≤25 ms unreachable" and "30 tok/s unreachable" (MTP K3 measured 33.0 ms) | | #51 | The untried even grids | | #61 | The EXL3 no-go, which is an estimate | | #63 | TP=1 MTP | | #46 | Told that the 0019 regression is fixed here | Also corrected: "one tt-metal process per host" is now **one per card** (`two-card-coexistence-20260925T0625Z`), and card locks and slots live only in the `tt-card-slot` skill. ## Branches and PRs - **Archived:** 31 DEAD remote branches were tagged `archive/<name>` and then deleted. Each was re-verified first: 27 are ancestors of master, and 4 squash-merged branches differ from master in 0 files. - **Re-homed:** the coordinator worktree's uncommitted #53/#55 files are now on the new branch `decode-campaign-53-55` (based on c76460b). The files are byte-identical and the worktree was not touched. - **Closed:** - #23, superseded by #66; - #16, branch merged; Ornith is now a 1-card candidate; - #15, content merged; the open goals moved to the new **#69** (1-card target: G4.3 context ladder, G4.4 accuracy, Step D bf4 coverage, and the reopened TP=1 items). - No ACTIVE branch was deleted. The SUPERSEDED branches (goal-4-nvfp4-single-card, ttsim-scout, dspark-acceptance-rate, thatch/lane2-serve-gotchas) were left in place. ## Migration notes for the active branches - **mtp-thinkingcap, thinkingcap-baseline:** rebase and drop the 28 pre-squash serving-throughput commits, e.g. `git rebase --onto origin/master <last duplicate>`. - Otherwise `patches/tt-metal/0021-0023` and the retired docs come back as modify/delete conflicts. - Take the deletions. Your ledger entry 0aaaaaaaaaaaaaaaaaa is in `docs/LEDGER.md` as `D-2026-09-24-thinkingcap`. - Your AGENTS.md edits are superseded by the new AGENTS.md. - **Branches that edit flake.nix or flake.lock** (agy/issue-49, -50, -64, tt-quant, bfp-quant): take master's inputs. - issue-50 should drop its tt-metal narHash hunk and use `nix develop ./nix/sim#runtime`. - `nodes.tt-metal.locked.rev` still exists; it is now `60053f1f`. - **bfpq and other arms that call `vllm-tt-serve.sh prod-argmax` with their own `attention-tp.py` via EXTRA_ENV:** the overlay is honoured, but `VERIFY=1` (the default) now blocks until the server is ready and refuses a missing banner. Pass `VERIFY=0` and call `verify-arm-banners.sh` yourself. - **Branches that edit `modules/tenstorrent-serving.nix`:** CI excludes it from nixfmt for now. Run nixfmt once in Phase B, after issue-45/49/50/64 land. - **agy/issue-36:** it edits the retired `UPSTREAM-WATCH-PLUGIN-METAL.md`. Fold its plugin-pin audit into `docs/BUILD.md`. - **agy/issue-38:** it adds `docs/LIFT-AUDIT-TT-METAL-56119.md`. Fold that into `docs/BUILD.md` or `REFERENCES.md`, not a new doc. - **repo-skills:** `perf-issue-advisor` references `docs/advisory/round2-logs`, which is now `bench/runs/archive/round2-logs-20260912`. - **decode-campaign-53-55:** rebase onto master after this PR. Its DIRECTIVES-LEDGER edit becomes a modify/delete conflict; take the deletion. ## Left for Phase B/C **Phase B:** - Land the active branches. - Consolidate the skills. - Rewrite the CLAUDE.md and retired-doc references in the frozen paths: comments in `bench/ab/*`, `modules/tenstorrent-serving.nix`, `patches/prod-local-metal`, and `patches/k|tt-metal`. - nixfmt the serving module. - Archive the frozen-path drivers. - Refresh `PROD-SERVING-SOURCES.md` after issue-64. **Phase C:** - `ttx` arm specs and per-card slots. - Turn k2-tracy into a build arg. - A tt-metal mirror to replace the numbered patches. ## Owner decisions needed 1. **Deploy the 0019 + mount fixes.** nixos-llm master still pins tt-stack `f4c3cda` (09-15). 2. **Enable the prefix arm in the unit (#48).** The module default is off. SERVING.md notes that the unit's `enablePrefixCaching` block-size/pool setup has no record of its own. 3. **The MTP identity-gap tolerance** (≤0.875 logit) and the batch-width tolerance. 4. **The ThinkingCap licence** (PolyForm Small Business) for commercial use. 5. **The #53 B=1 ≤25 ms gate** versus makespan ranking. 6. **Whether to keep tt-model-publish's unpushed local commit.** 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Grok added 2 commits 2026-09-25 09:43:59 +02:00

fix(serving): re-apply patch 0019, mount attention/tp.py, prod-true sweep baseline ... 7e942d0662

What was believed: production and every "prod" bench arm run the recorded source set,
with the prefilled-slots fix (0019) and SDPA streaming (0020) on, and the standard
sweep's baseline is production.

What the evidence says (verified 2026-09-25 by diffing 583d1c1..89f75ce on
qwen36_vllm.py):
- 89f75ce ("complete unification of prefix caching...") deleted the 0019 block from
  prefill_forward. Nothing mounted read QWEN36_PREFILLED_SLOTS_FIX afterwards, yet the
  unit kept setting it and never checked a banner, so the stale-slot token leak under
  device argmax (lever1b/lever1-argmax-slotfix records) was back on master silently.
  Re-applied the recorded patch verbatim (it applies forward cleanly) ahead of the
  prefix branch, and the unit's readiness probe now refuses a server without the
  "QWEN36_PREFILLED_SLOTS_FIX engaged" banner (the ping it already sends triggers it).
- scripts/vllm-tt-serve.sh never mounted attention/tp.py, the only file that reads
  QWEN36_SDPA_STREAMING, and plain `prod` skipped qwen36_vllm.py/generator.py/
  model_runner.py. Every prod profile now mounts exactly the unit's seven files (flat
  PATCHED_DIR or this checkout's patches/ tree), skips a destination an EXTRA_ENV -v
  overlay already covers, and by default (VERIFY=1) blocks until a ping succeeds and
  refuses unless every QWEN* flag set to 1 in the container's effective env shows its
  banner (new scripts/verify-arm-banners.sh). The pre-unification prefix extras are
  opt-in via PREFIX_LEGACY_MOUNTS=1 because the unit does not mount them.
- bench/ab/sweep-32k.sh's "baseline" was goal1 plus a hand-copied 09-14 env (bf16 KV,
  host sampling, no bf4 down, no streaming, no 0019). Baseline is now the prod-argmax
  profile, i.e. the unit; the second arm is "candidate" = baseline + CAND_ENV. Both arms
  are identity-checked before anything is recorded. The reporter reads both the old
  "optimized" and the new "candidate" arm names.

Nothing was deployed; cfx-llm2 is untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ci: public tt-metal pin, Nix ttnn moved to a host-local sub-flake, pinned formatter ... 987e88dde4

What was believed: CI was red for formatting only, and the Nix tt-metal/ttnn build
was part of the stack.

What the evidence says:
- Past the nixfmt failure, nothing depending on tt-metal evaluated anywhere: the
  input was git+file:///home/orchestra/ttm-build/src (main@60053f1f + #55548 + cfx),
  unfetchable off-host and with a stale narHash even on it. It would also have
  landed in nixos-llm's lock on the next bump.
- The Nix ttnn is cp314; the serving image is cp310, so it can never reach
  production. nixos-llm (origin/master 8ba1e9d) imports only
  nixosModules.tenstorrent-{host,serving}. The only live consumer is the ttsim
  correctness harness on the orchestra host (agy/issue-50 used it on 09-25).

So: `tt-metal` is now public upstream 60053f1f (the base of the local merge;
tt_metal/soc_descriptors is byte-identical, so ttsim is unchanged and
ttsim-smoke passes). The tt-metal/ttnn/sfpi build and the `runtime` shell moved to
nix/sim (`nix develop ./nix/sim#runtime`), which CI and nixos-llm never evaluate;
its ttnn evaluates to the SAME store path that was already built
(mkznmm2f...-python3.14-ttnn-0.77.0-dev20260813-76983d0), so the sim harness loses
nothing. Dropped the inputs and packages nothing consumes: tt-inference-server +
probes.nix, vllm-tt-plugin (only copied into the recipe, at the wrong rev),
tt-lang, tt-vscode-toolkit, tt-cli, tt-model-manager (+3 opentelemetry
derivations), and production-serving-recipe.nix, whose patch list stopped at 0017
and whose hashes matched no served file. nixosModules names and every option are
unchanged.

CI now takes nixfmt/shellcheck from the flake-locked nixpkgs instead of the
runner's floating registry. modules/tenstorrent-serving.nix is excluded from the
format check until the four agy branches that edit it land (nixfmt re-indents its
whole embedded probe script). Locally: nixfmt --check, shellcheck scripts/*.sh,
`nix flake check` (ttsim-bh, ttsim-bh-x2, tenstorrent-host VM test) and
`nix run .#ttsim-smoke` all pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Grok added 1 commit 2026-09-25 09:47:26 +02:00

chore: delete dead images, patches and drivers; archive what records cite ... 93c9f7344e

What was believed: patches/ is the authoritative, applied patch series; docker/
holds the production image lineage; bench/ and scripts/ are the live harness.

What the evidence says (audit-build/audit-bench, re-verified 2026-09-25 with
`patch --dry-run` against upstream HEAD, the flake pin and the prod mount, and a
reference grep over master plus every active branch):
- tt-metal 0001/0002/0009/0010/0013, the duplicate pr53587 diff and the P150X2
  tt-inference-server patch apply to nothing. The block-keyed prefix series
  (plugin 0002-0006, patches/vllm, patches/prefix-tests) is dead: prefix caching
  shipped vLLM-native and the unit mounts none of it (prefix-tests already read a
  gitignored tree; bench/acceptance_negative_gates.py already degrades to an error
  dict without it).
- 0012, 0014-0018, 0020 and plugin 0001 are 93-100% present in patches/prod-local-*;
  patches/README.md now records which file absorbed each. 0019 stays (just
  re-applied), and 0021-0023 stay until mtp-thinkingcap / thinkingcap-baseline
  rebase, because their pre-squash history re-adds them.
- Dockerfile.vllm-tt / -overlay are superseded by -src; -k34m/-k5/-k5rf and the
  K3-K5 scripts in patches/k are levers under "op-count reduction, closed by
  measurement" and in no image. k2-tracy is kept (the only per-op device profile
  path); turning it into a build arg belongs to the phase C image work.
- generated/ (tt-metal runtime output), bench/wip (resident decode landed),
  bench/matrix, the unlocked lever*-ab.sh / prefill-sweep.sh, mem-*.sh,
  m4-tp2-generate.sh, tests/run_tp_contract.py and the 7-line root
  BLACKHOLE_GUIDE.md stub have zero live referrers.
- docs/advisory/round2-logs is raw evidence, so it MOVED to
  bench/runs/archive/round2-logs-20260912 instead of being deleted.
- bench/fixures (typo) folded into bench/fixtures/dspark; the one code referrer
  (bench/probes/dspark-full-drafter-cpu-tp.py) is repointed.
- tt-bench.py, tt-sweep.py, long-context-bench.py, prefill-bench.py and
  run-hardware-server.sh are cited by records, so they moved to bench/archive/.
  Closed-work drivers under the frozen bench/ab and bench/probes wait for phase B.

Everything deleted is recoverable from 682dd38.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hermes commented 2026-09-25 09:51:49 +02:00

Owner

Copy link

verdict: approve
tier: T1 (CI/lock hygiene; small ttnn patch correctness fix). Note: PR body is still the "work in progress" placeholder ("full body filled in when phase A is complete") and says "Do not merge" — I approve the direction of this phase-A chunk, not a merge yet.

rationale: This is a clean WIP cleanup. It fixes a real correctness gap in tt-metal's ttnn qwen36 prefill (records prefilled slots for the decode reset — previously never written on the Qwen path), removes dead local-only git+file inputs that broke CI/nix flake check/nixos-llm locks, and adds a fail-closed arm identity check for the A/B serving sweep. Verified: no live git+file refs remain in the tree (only a comment mentions the old path); nix/sim still imports ../ttnn.nix and ../sfpi.nix, which still exist; tests/tenstorrent-host.nix present. CI has_actions: true, so the label IS an enforcement gate.

concerns:

• [warn] The ttsim correctness harness is now built only from the host-local sub-flake nix/sim (nix develop ./nix/sim#runtime on the orchestra host). It is deliberately kept out of the root flake, so CI/nix flake check/nixos-llm lock never evaluate it — that's correct and keeps the local tt-metal-full rev off the public lock. Confirm the orchestra host actually has develop ./nix/sim#runtime building green; if it silently doesn't, the ttsim correctness harness is gone from every path except a hand-run on one box.
• [warn] flake.lock now pins tt-metal at 60053f1f (upstream public main), whereas the pre-PR tt-metal/tt-metal-full pointed at a local trymerge merge of main + PR #55548 (MTP spec decode). The diff argues the simulators are byte-identical for this (SoC descriptors unchanged) and production never consumed the local merge. If any ttsim number depended on PR #55548's changes, that would be a regression — worth one confirming line in the phase-A body.
• [warn] CI Compare pinned revs loop is now ttsim tt-metal only; the tt-inference-server/vllm-tt-plugin/tt-lang comparisons were dropped. Fine since those inputs were removed, but document it so a future nix flake update doesn't resurrect a comparison against a now-dead input.
• [nit] .forgejo/workflows/ci.yml excludes modules/tenstorrent-serving.nix from nixfmt until a later branch lands — fine, but that exception comment (agy/issue-45/49/50/64) should reference a real tracking issue if one exists.

No secret-leak / vuln / red-flag matches. The serving script changes are fail-closed by design (refuse to record a mis-armed server), which is the correct pattern.

**verdict:** approve **tier:** T1 (CI/lock hygiene; small ttnn patch correctness fix). Note: PR body is still the "work in progress" placeholder ("full body filled in when phase A is complete") and says "Do not merge" — I approve the *direction* of this phase-A chunk, not a merge yet. **rationale:** This is a clean WIP cleanup. It fixes a real correctness gap in tt-metal's ttnn qwen36 prefill (records prefilled slots for the decode reset — previously never written on the Qwen path), removes dead local-only `git+file` inputs that broke CI/`nix flake check`/nixos-llm locks, and adds a fail-closed arm identity check for the A/B serving sweep. Verified: no live `git+file` refs remain in the tree (only a comment mentions the old path); `nix/sim` still imports `../ttnn.nix` and `../sfpi.nix`, which still exist; `tests/tenstorrent-host.nix` present. CI `has_actions: true`, so the label IS an enforcement gate. **concerns:** - [warn] The `ttsim` correctness harness is now built only from the host-local sub-flake `nix/sim` (`nix develop ./nix/sim#runtime` on the orchestra host). It is deliberately kept out of the root flake, so CI/`nix flake check`/nixos-llm lock never evaluate it — that's correct and keeps the local `tt-metal-full` rev off the public lock. Confirm the orchestra host actually has `develop ./nix/sim#runtime` building green; if it silently doesn't, the ttsim correctness harness is gone from every path except a hand-run on one box. - [warn] `flake.lock` now pins `tt-metal` at `60053f1f` (upstream public main), whereas the pre-PR `tt-metal`/`tt-metal-full` pointed at a local `trymerge` merge of main + PR #55548 (MTP spec decode). The diff argues the simulators are byte-identical for this (SoC descriptors unchanged) and production never consumed the local merge. If any ttsim number depended on PR #55548's changes, that would be a regression — worth one confirming line in the phase-A body. - [warn] CI `Compare pinned revs` loop is now `ttsim tt-metal` only; the `tt-inference-server`/`vllm-tt-plugin`/`tt-lang` comparisons were dropped. Fine since those inputs were removed, but document it so a future `nix flake update` doesn't resurrect a comparison against a now-dead input. - [nit] `.forgejo/workflows/ci.yml` excludes `modules/tenstorrent-serving.nix` from nixfmt until a later branch lands — fine, but that exception comment (agy/issue-45/49/50/64) should reference a real tracking issue if one exists. No secret-leak / vuln / red-flag matches. The serving script changes are fail-closed by design (refuse to record a mis-armed server), which is the correct pattern.

Grok added 1 commit 2026-09-25 10:01:38 +02:00

docs: 104 docs -> 11 plus a lean AGENTS.md; only LEDGER may call something closed ... e7bfc6da7b

Before: the ledger was the authority for "closed by measurement, do not
re-propose", but the audit found that 22 of 34 closures were estimates,
code-reads, third-party numbers, or no longer true. Some examples:
- MTP was "dropped, never measured" and is now x1.4-1.6 at B=1.
- "bf4 MLP retired" contradicted both the owner's 09-15 "keep BF4" and the
  G2 accuracy result.
- The "one process per host" rule fails for single-card processes.
- "Prefix caching not deployable" was written before it landed on 09-18.

About 20 docs still told agents to pass --no-enable-prefix-caching. CLAUDE.md
was cited in more than 20 places after it was deleted. DOC-STATUS existed only
to list which of the other 100 docs were stale.

After:
- docs/LEDGER.md: dated directive and result IDs.
  - It adds the 09-15..09-25 entries: keep-BF4, the RAM ruling, prefix caching,
    SDPA streaming, argmax on the prefix arm, phase2a with honest numbers,
    ThinkingCap and MTP, the two-card coexistence, GPTQ G2, and the 0019
    regression.
  - A Closed-claims table gives each closure its class, scope and record.
  - An explicit Reopened list covers every (b)/(d) item.
  - The DSpark block is dropped, and the missing 134735Z citation is fixed
    to 134045Z.
- docs/STATUS.md: per-TP, per-B scoreboard (PP, TG, TTFT, makespan), each
  cell with its record.
  - It says what nixos-llm master actually pins: f4c3cda, 09-15, with prefix
    caching off.
  - The prefix-arm table is labelled as an arm, not as production.
- GOALS, DEV-LOOP, BUILD, HARDWARE, SERVING, PREFILL, MTP, QUANT and
  REFERENCES fold in everything still true from the retired docs.
  specs/p150x1 is new.
- AGENTS.md is rewritten at 155 lines.
  - It covers the 1-card and 2-card targets and the "prove it or it's open"
    rule.
  - It keeps the hard safety rules.
  - It keeps only the traps with records: k_chunk_size=32, LoFi+fp32_dest_acc,
    batch width.
  - The card rule is now one process per card, with the record cited.
  - Card locks and slots live only in the tt-card-slot skill.
- autoresearch.sh, its measure script and its test are removed.
  - They printed docs/DSPARK-NEXT.md for a DSpark autoresearch loop that the
    old ledger itself recorded as off-objective.
- Code comments that pointed at retired docs now point at their successor,
  as "docs/NEW.md (was OLD.md)", in the non-frozen files. Frozen paths are
  untouched until their branches land: bench/ab, bench/probes,
  modules/tenstorrent-serving.nix, patches/.

Every retired doc can be recovered with `git show 682dd38:docs/<name>`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Grok added 1 commit 2026-09-25 10:04:14 +02:00

bench/runs: archive 182 uncited records, add INDEX.tsv and a CI check that citations resolve ... 1aaaee7b19

Before: 373 files sat flat in bench/runs, and nothing enforced "never lose
evidence". The rewritten docs and branch drafts both cited records that did
not exist:
- the old ledger's target-shape-ttft-B4-256k-20260914T134735Z (the real file
  is 134045Z);
- prefill-audit-ab-down-minimal-20260912T110412Z.
About half of the records were no longer cited by anything.

After:
- scripts/check-cited-records.py (new, runs in CI) fails if a bench/runs path
  or a timestamped record stem cited in AGENTS.md, README.md, docs/, specs/,
  bench/README.md, patches/README.md or .claude/skills is missing.
  - `<branch>:bench/runs/...` and `[branch]` citations are skipped; they land
    with their branch.
  - A bare `bench/runs/*.jsonl` is treated as the naming convention, not as a
    citation.
- A record stays live if any of these cite it:
  - any master file outside bench/runs (docs, scripts, skills, code);
  - any file an ACTIVE branch adds or changes;
  - the coordinator's uncommitted #53/#55 decode-campaign files;
  - a kept record (one hop of evidence chains).
  Paired .log, .table.txt and similar siblings stay beside their jsonl.
  That leaves 183 records live.
- Moved 161 uncited records to bench/runs/archive/<YYYY-MM>/ and 21 boot-fail
  logs (zero-byte or orphaned) to archive/failed/. The failed logs are kept as
  negative evidence.
- Deleted 8 zero-byte, uncited .jsonl files, which hold no bytes.
- bench/runs/archive/MOVED.tsv maps every old path to its new path, so older
  commits and docs that cite an archived record can still find it.
- bench/runs/INDEX.tsv (`--index`) lists every record: live or archived, size,
  first-line kind, and which master files cite it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Grok referenced this pull request 2026-09-25 10:06:46 +02:00

bfp-quant: EXL3-style Hessian-aware offline quantizer → native bfp4_b (grid-snapped), + weight accuracy gate #62

Grok referenced this pull request 2026-09-25 10:06:46 +02:00

DSpark drafter (ThatchCloud) on ThinkingCap #66

Grok referenced this pull request 2026-09-25 10:06:47 +02:00

hyp D: non-matmul decode ops share the matmul Tensix grid, so they cannot hide under the weight read #56

Grok referenced this pull request 2026-09-25 10:06:47 +02:00

prefill: widen MLP gate|up AGMM and GDN in-proj grids from 8x9 to 11x10 full Blackhole mesh #51

Grok referenced this pull request 2026-09-25 10:06:47 +02:00

research: exl3xpu / EXL3 trellis quant on Blackhole — port is no-go; EXL3-style offline bf4 quantizer + accuracy gate instead #61

Grok referenced this pull request 2026-09-25 10:06:47 +02:00

MTP on ThinkingCap-Qwen3.8-27B-FP8 (TP=2): inventory, per-row B=4/8 design, go/no-go, phased gates #63

Grok referenced this pull request 2026-09-25 10:06:47 +02:00

generator: device decode position buffer contains uninitialized values alongside valid ones #46

Grok added 1 commit 2026-09-25 10:08:01 +02:00

docs: point the 1-card items at the new issue #69 ... 347a3cbda3

The ledger and scoreboard pointed at "the 1-card issue" before it existed. It
is now #69, which carries PR #15's open G4.3/G4.4/Step D plus the reopened
TP=1 items.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Grok merged commit 347a3cbda3 into master 2026-09-25 10:09:41 +02:00

Grok commented 2026-09-25 10:09:43 +02:00

Author

Owner

Copy link

agent-gateway closeout (merged): cleared agent:* labels.
Hermes kanban card (if any) is left for Hermes to archive/complete.

agent-gateway closeout (merged): cleared `agent:*` labels. Hermes kanban card (if any) is left for Hermes to archive/complete.

Grok referenced this pull request from a commit 2026-09-25 10:16:31 +02:00

Merge origin/master (harness overhaul, PR #68) into mtp-thinkingcap

Grok referenced this pull request from a commit 2026-09-25 10:22:34 +02:00

Merge origin/master (PR #68 harness overhaul) into tt-quant

Grok referenced this pull request 2026-09-25 10:31:54 +02:00

Quantize on TT cards: CUDA-free GPTQ/LDLQ bfp4 pipeline (Hessians, GPTQ sweep, FP8 teacher on Blackhole) #67

Grok referenced this pull request from a commit 2026-09-25 10:48:51 +02:00

docs: move the MTP G2 result and the TP=1 directive into LEDGER/MTP/STATUS

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

human-approved

Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels

human-approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack!68

No description provided.