bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

bfp-quant: EXL3-style Hessian-aware offline quantizer → native bfp4_b (grid-snapped), + weight accuracy gate #62

New issue
Open
opened 2026-09-24 11:11:53 +02:00 by Grok · 8 comments
Grok commented 2026-09-24 11:11:53 +02:00
Owner
Copy link

Tracking issue for the recommended path out of #61: take EXL3's quantization method (Hessian-aware LDLQ/GPTQ rounding, optional incoherence rotation), not its trellis bitstream, and emit weights that sit exactly on the native bfp4_b grid. The Tensix unpacker reads them for free, so there is no decode cost and no runtime or kernel change.

Why

  • EXL3's trellis codes aren't a hardware format. On-device decode is an estimated 25–50 ms/step on the SFPU, against a ≤4–6 ms/step byte saving (ESTIMATE, #61).
  • What makes EXL3 valuable is rounding quality per bit. We can take that offline.
  • Payoff (ESTIMATE):
    • Better quality on the MLP, which is already bf4, at the same bytes.
    • If the gate passes, attention and GDN projections move from bf8 to bf4: −2.6 to −4.0 ms/step and +3.4 GiB card memory (~110k bf8 KV tokens).
  • Production's bf4 down_proj (QWEN36_MLP_DOWN_BF4=1) has no accuracy record at all. goal3-down-proj-bf4-2026-09-05.jsonl says "unverified".

Phases (details and go/no-go in #61 "Integration plan")

  • P0 — offline error study (no cards).
    • Exact Python emulation of the tt-metal host bfp4_b/bfp8_b packer (convert_u32_to_bfp), bit-verified against ttnn host APIs.
    • Collect calibration Hessians for Qwen3.8-27B bf16. This needs an off-box CUDA GPU: cfx-llm2 has ~5.5 GB of probe RAM.
    • Compute the proxy error tr(E H Eᵀ)/tr(W H Wᵀ) per linear group for:
      • RTN bf4 from FP8 and from bf16
      • RTN bf8
      • LDLQ→bf4 grid
      • LDLQ + rotation
      • EXL3 4.0/3.0 bpw (via exl3xpu ref.py)
    • Go: LDLQ→bf4 ≥2× lower error than RTN bf4 on attn/GDN, and within 2× of EXL3-4bpw. Stop: <25% gain everywhere.
  • P1 — accuracy gate, built once and reused.
    • Teacher-forced KL / top-1 vs bf16, measured in the last 256 tokens of 4k/32k/128k contexts.
    • Harder keyed retrieval, the agentic tool-call canary, and a B=4/B=8 fixed-width reproducibility check.
    • Not short-prompt GSM8K.
    • First deliverable: the missing accuracy record for prod's bf4 MLP.
  • P2 — grid-snapped MLP checkpoint.
    • bf16 safetensors whose values are exactly on the bfp4_b grid, so the host packer is the identity. 100% bitwise round-trip offline.
    • Distinct tensor_cache_* suffix: a stale cache silently serves old weights.
    • Predicted Δ decode 0 ± 0.5 ms. Gate: KL strictly below prod RTN at every context.
    • Packaged as nix run .#bfp-quant.
  • P3 — attention/GDN → bf4.
    • Per-group dtype policy env var, banner-logged.
    • Prediction −2.6 to −4.0 ms/step; falsifier <1.0 ms.
    • Measured as makespan/TTLT with bench/ab/sweep-32k.sh plus a prod-weight-mix control arm, interleaved.
  • P4 (optional) — rotation (E2) and per-tile mixed bfp8/bfp4/bfp2 via MatmulCustomCompressed (E3). E3 is blocked on upstream #42841/#42586.

Rules

  • Every probe fails closed and writes bench/runs/*.jsonl.
  • Card holds are serialised on /tmp/ttlock, with MemAvailable ≥4 GB checked before every boot.
  • Arms self-identify: dtype per group plus checkpoint hash in the banner.
  • Label everything MEASURED / THIRD-PARTY / ESTIMATE.

Effort (ESTIMATE): P0–P3 ≈ 5–8 engineer-weeks, 8–10 card holds.

Refs: #61 (research), #60.

Tracking issue for the recommended path out of #61: take EXL3's **quantization method** (Hessian-aware LDLQ/GPTQ rounding, optional incoherence rotation), not its trellis bitstream, and emit weights that sit **exactly on the native `bfp4_b` grid**. The Tensix unpacker reads them for free, so there is no decode cost and no runtime or kernel change. ## Why - EXL3's trellis codes aren't a hardware format. On-device decode is an estimated 25–50 ms/step on the SFPU, against a ≤4–6 ms/step byte saving (ESTIMATE, #61). - What makes EXL3 valuable is **rounding quality per bit**. We can take that offline. - **Payoff (ESTIMATE):** - Better quality on the MLP, which is already bf4, at the same bytes. - If the gate passes, attention and GDN projections move from bf8 to bf4: **−2.6 to −4.0 ms/step** and **+3.4 GiB card memory** (~110k bf8 KV tokens). - **Production's bf4 down_proj (`QWEN36_MLP_DOWN_BF4=1`) has no accuracy record at all.** `goal3-down-proj-bf4-2026-09-05.jsonl` says "unverified". ## Phases (details and go/no-go in #61 "Integration plan") - [ ] **P0 — offline error study (no cards).** - Exact Python emulation of the tt-metal host `bfp4_b`/`bfp8_b` packer (`convert_u32_to_bfp`), bit-verified against ttnn host APIs. - Collect calibration Hessians for Qwen3.8-27B **bf16**. This needs an off-box CUDA GPU: cfx-llm2 has ~5.5 GB of probe RAM. - Compute the proxy error `tr(E H Eᵀ)/tr(W H Wᵀ)` per linear group for: - RTN bf4 from FP8 and from bf16 - RTN bf8 - LDLQ→bf4 grid - LDLQ + rotation - EXL3 4.0/3.0 bpw (via exl3xpu `ref.py`) - **Go:** LDLQ→bf4 ≥2× lower error than RTN bf4 on attn/GDN, and within 2× of EXL3-4bpw. **Stop:** <25% gain everywhere. - [ ] **P1 — accuracy gate, built once and reused.** - Teacher-forced KL / top-1 vs bf16, measured in the last 256 tokens of 4k/32k/128k contexts. - Harder keyed retrieval, the agentic tool-call canary, and a B=4/B=8 fixed-width reproducibility check. - Not short-prompt GSM8K. - First deliverable: the missing accuracy record for prod's bf4 MLP. - [ ] **P2 — grid-snapped MLP checkpoint.** - bf16 safetensors whose values are exactly on the bfp4_b grid, so the host packer is the identity. 100% bitwise round-trip offline. - Distinct `tensor_cache_*` suffix: a stale cache silently serves old weights. - Predicted Δ decode 0 ± 0.5 ms. Gate: KL strictly below prod RTN at every context. - Packaged as `nix run .#bfp-quant`. - [ ] **P3 — attention/GDN → bf4.** - Per-group dtype policy env var, banner-logged. - Prediction −2.6 to −4.0 ms/step; **falsifier <1.0 ms**. - Measured as makespan/TTLT with `bench/ab/sweep-32k.sh` plus a prod-weight-mix control arm, interleaved. - [ ] **P4 (optional)** — rotation (E2) and per-tile mixed bfp8/bfp4/bfp2 via `MatmulCustomCompressed` (E3). E3 is blocked on upstream #42841/#42586. ## Rules - Every probe fails closed and writes `bench/runs/*.jsonl`. - Card holds are serialised on `/tmp/ttlock`, with `MemAvailable` ≥4 GB checked before every boot. - Arms self-identify: dtype per group plus checkpoint hash in the banner. - Label everything MEASURED / THIRD-PARTY / ESTIMATE. Effort (ESTIMATE): P0–P3 ≈ 5–8 engineer-weeks, 8–10 card holds. Refs: #61 (research), #60.
Grok commented 2026-09-24 12:14:36 +02:00
Author
Owner
Copy link

P0 progress (branch bfp-quant, 3 commits, pushed, no PR). No card time used.

Everything below ran on host orchestra, which has no /dev/tenstorrent. The only access to cfx-llm2 was read-only scp of checkpoint shards.

What was built (tools/bfp-quant/)

  • bfp.py: exact emulation of the tt-metal host packer for bfp8_b and bfp4_b (convert_u32_to_bfp, pack_as_bfp_tiles, and the unpackers). It provides snap_to_bfp4(w) and check_on_grid (fails closed).
  • Reference packer: packer-harness/. The verbatim upstream blockfloat_common.cpp, bfloat4.cpp and bfloat8.cpp at the flake-pinned rev 76983d00, compiled standalone against a shim that replaces only logging, tracy, MetalContext and Tile.
    • I did not use ttnn.from_torch(device=None). Its host path constructs MetalContext, which constructs tt::Cluster, which opens /dev/tenstorrent on a card host.
    • Also, nix build .#ttnn currently fails eval with a narHash mismatch on the local tt-metal input.
  • Packer semantics that matter for the quantizer:
    • The shared exponent is the max raw fp32 exponent of the 16 values, chosen before rounding.
    • A block max that rounds past the binade saturates (1.9 becomes 1.75). The exponent is not bumped.
    • Bits shifted below the mantissa are dropped (no sticky bit). Rounding is RNE.
    • Subnormal inputs become 0, and -0 loses its sign.
    • Group layout: ttnn stores linears as W^T [in, out]. So one shared exponent covers 16 consecutive output channels at one input index, and every bfp block sits inside a single GPTQ column.
    • The prod transforms (tile-pair swiglu interleave, TP shard on out/in) all move data in 32-aligned units, so a snapped weight stays snapped through them.
  • quant.py: RTN and GPTQ/LDLQ onto the bfp grid.
    • How the exponent choice is guaranteed consistent: each column is rounded with the packer's own arithmetic at the moment it is quantized. The rounded block keeps its max exponent, so the packer re-derives the same shared exponent.
    • Optional per-block exponent search {e, e+1} also stays exact: values on the coarser grid are also on the grid the packer will pick.
    • Both claims are checked on every output, not assumed.
  • bfp_quant.py quantize writes grid-snapped bf16 safetensors for the tensors that --include selects. RTN works without Hessians; GPTQ takes --hessians DIR. Untouched shards are symlinked.
    • It also writes a manifest with per-tensor sha256 and a suggested distinct tensor_cache suffix, to avoid the stale-cache trap.
    • The source checkpoint and the tensor selection are parameters. There are no hard-coded Qwen paths or names. Which modules are bf16 is read from the data (dtype plus presence of a scale tensor), not from modules_to_not_convert / quantization_config.ignore.
  • proxy_study.py: streamed, per-tensor, Hessian-free error map. It also has --remote host:/dir to copy one shard at a time.
  • nix run .#bfp-quant -- {test,verify-packer,synthetic,proxy,quantize} (nix/bfp-quant.nix). Tested end to end.

Verification (all MEASURED)

  • bench/runs/bfp-packer-emulation-20260924T093051Z.jsonl: 68 cases x {bfp8_b, bfp4_b}.
    • Packed words and unpacked values are bitwise identical to the real code. This holds for bf16 input in face order (the exact production call) and for fp32 input in row-major order.
    • Adversarial cases covered: binade edge 1.9 and bf16-max, exact ties for both formats, exp_diff > 24 and > 31, zeros and -0, subnormal inputs, the bottom of the exponent range, huge dynamic range, all-negative.
    • Real weights covered: Qwen3.5-9B-FP8 (MLP, GDN, attention, lm_head) and a Qwen3.8-27B-FP8 GDN layer. 374M values in total.
    • Idempotence: snap, then real pack, then real unpack returns snap on 100% of values, bitwise, in every case.
    • The negative-control mutants (exponent bump, truncation, half-away rounding) were all detected. An independent float reimplementation agrees exactly.
  • bench/runs/bfp-packer-emulation-20260924T093819Z.jsonl: a snapped 9B gate/up checkpoint written by quantize is a fixed point of the REAL packer. The unsnapped down_proj fails the same --expect-snapped gate, as it must.
  • bench/runs/bfp-quant-synthetic-20260924T094254Z.jsonl: synthetic only. GPTQ beats RTN on tr(E H Eᵀ)/tr(W H Wᵀ) in every synthetic layer, by 16-23x.
    • That size of gain comes from a steep synthetic spectrum with outlier channels. It is not a prediction for Qwen.
    • Unit test: GPTQ with H = I reduces exactly to RTN.
  • bench/runs/bfp-proxy-hfree-20260924T092920Z.jsonl: Qwen3.8-27B-FP8, layers 0-3, 30, 31, 62, 63 plus embed and lm_head. Hessian-free proxy (H = I), measured against the FP8 weights dequantised the way prod does.
    • RTN bfp4 relMSE is 1.31-1.47e-2 in every group. The lowest groups are MLP gate/up/down; the highest are GDN out_proj and attention v_proj.
    • RTN bfp8 is 5.5-6.4e-5, about 235x lower.
    • 8-10% of bfp4 blocks saturate at the block max.
    • The {e, e+1} exponent search gains at most 0.1%, so it is not worth having.
    • The groups differ very little because bfp4 has roughly constant relative precision. Ranking groups for P3 needs Hessians and P1, not weight statistics. This proxy also cannot rank LDLQ, since GPTQ with H = I is RTN.

Compute findings and blockers

  • No suitable calibration compute is known.
    • cfx-llm0 has 4x RTX 3080 10 GB. It is production vLLM serving 27B at TP=4, so using it means taking cluster inference down, which needs owner sign-off.
    • nixos (and orchestra, a container on it) has 47 GB RAM, 20 cores and an AMD 6900 XT (ROCm, no CUDA). It is the build/gaming host.
    • cfx-llm2 has 15 GB RAM and 4 cores. cfx-llm4 is Vulkan AMD. cfx-mgmt has a 6 GB 2060.
    • No host is marked for batch ML work.
    • Options: rent an A100/H100 or a CUDA box with at least 128 GB RAM; get approval for a cfx-llm0 window; or run a slow layer-streamed CPU calibration on nixos, which would load the shared host.
  • No bf16 checkpoint anywhere. Only Qwen/Qwen3.8-27B-FP8 exists, on cfx-llm2 at /var/lib/models/hf/hub/models--Qwen--Qwen3.8-27B-FP8, snapshot 017b9c7a. So bf4-from-bf16 vs bf4-from-FP8 is not measured. orchestra has 20 GB of free disk and cannot stage 54 GB.
  • Target checkpoint change (owner): calibration and error studies should target bottlecapai/ThinkingCap-Qwen3.8-27B bf16 (FP8 variant: -FP8) once gated access is granted. Nothing in the tool assumes Qwen paths or the name of the bf16-kept-modules key.
  • EXL3 comparison: exl3xpu ref.py (MIT) only reconstructs tensors from an existing EXL3 checkpoint. It needs an EXL3 4.0/3.0 bpw checkpoint of the same base, plus the bf16 reference, to be meaningful, so it is not done yet.

Next step

  1. Get GPU access or approval, and bf16 (ThinkingCap) access.
  2. Run exllamav3-style Hessian collection (sc_trace-style self-sampled calibration), saving one H per module as <module>.safetensors, which quantize --hessians already reads.
  3. Compute proxy arms (i)-(iii) per group, check them against the go/no-go criteria, then add (v) EXL3 via ref.py.

P0 checkbox is left unticked: the emulation and quantizer parts are done, but the Hessian-weighted study is blocked on compute.

## P0 progress (branch `bfp-quant`, 3 commits, pushed, no PR). No card time used. Everything below ran on host `orchestra`, which has no `/dev/tenstorrent`. The only access to cfx-llm2 was read-only `scp` of checkpoint shards. ### What was built (`tools/bfp-quant/`) - **`bfp.py`: exact emulation of the tt-metal host packer** for bfp8_b and bfp4_b (`convert_u32_to_bfp`, `pack_as_bfp_tiles`, and the unpackers). It provides `snap_to_bfp4(w)` and `check_on_grid` (fails closed). - **Reference packer: `packer-harness/`.** The verbatim upstream `blockfloat_common.cpp`, `bfloat4.cpp` and `bfloat8.cpp` at the flake-pinned rev `76983d00`, compiled standalone against a shim that replaces only logging, tracy, `MetalContext` and `Tile`. - I did not use `ttnn.from_torch(device=None)`. Its host path constructs `MetalContext`, which constructs `tt::Cluster`, which opens `/dev/tenstorrent` on a card host. - Also, `nix build .#ttnn` currently fails eval with a narHash mismatch on the local tt-metal input. - **Packer semantics that matter for the quantizer:** - The shared exponent is the max raw fp32 exponent of the 16 values, chosen **before** rounding. - A block max that rounds past the binade **saturates** (1.9 becomes 1.75). The exponent is not bumped. - Bits shifted below the mantissa are dropped (no sticky bit). Rounding is RNE. - Subnormal inputs become 0, and -0 loses its sign. - **Group layout:** ttnn stores linears as W^T [in, out]. So one shared exponent covers **16 consecutive output channels at one input index**, and every bfp block sits inside a single GPTQ column. - The prod transforms (tile-pair swiglu interleave, TP shard on out/in) all move data in 32-aligned units, so a snapped weight stays snapped through them. - **`quant.py`: RTN and GPTQ/LDLQ onto the bfp grid.** - How the exponent choice is guaranteed consistent: each column is rounded with the packer's own arithmetic at the moment it is quantized. The rounded block keeps its max exponent, so the packer re-derives the same shared exponent. - Optional per-block exponent search {e, e+1} also stays exact: values on the coarser grid are also on the grid the packer will pick. - Both claims are **checked on every output**, not assumed. - **`bfp_quant.py quantize`** writes grid-snapped bf16 safetensors for the tensors that `--include` selects. RTN works without Hessians; GPTQ takes `--hessians DIR`. Untouched shards are symlinked. - It also writes a manifest with per-tensor sha256 and a suggested distinct `tensor_cache` suffix, to avoid the stale-cache trap. - The source checkpoint and the tensor selection are parameters. There are no hard-coded Qwen paths or names. Which modules are bf16 is read from the data (dtype plus presence of a scale tensor), not from `modules_to_not_convert` / `quantization_config.ignore`. - **`proxy_study.py`**: streamed, per-tensor, Hessian-free error map. It also has `--remote host:/dir` to copy one shard at a time. - **`nix run .#bfp-quant -- {test,verify-packer,synthetic,proxy,quantize}`** (`nix/bfp-quant.nix`). Tested end to end. ### Verification (all MEASURED) - `bench/runs/bfp-packer-emulation-20260924T093051Z.jsonl`: 68 cases x {bfp8_b, bfp4_b}. - Packed words and unpacked values are **bitwise identical** to the real code. This holds for bf16 input in face order (the exact production call) and for fp32 input in row-major order. - Adversarial cases covered: binade edge 1.9 and bf16-max, exact ties for both formats, exp_diff > 24 and > 31, zeros and -0, subnormal inputs, the bottom of the exponent range, huge dynamic range, all-negative. - Real weights covered: Qwen3.5-9B-FP8 (MLP, GDN, attention, lm_head) and a Qwen3.8-27B-FP8 GDN layer. 374M values in total. - **Idempotence:** snap, then real pack, then real unpack returns snap on 100% of values, bitwise, in every case. - The negative-control mutants (exponent bump, truncation, half-away rounding) were all detected. An independent float reimplementation agrees exactly. - `bench/runs/bfp-packer-emulation-20260924T093819Z.jsonl`: a snapped 9B gate/up checkpoint written by `quantize` is a **fixed point of the REAL packer**. The unsnapped down_proj fails the same `--expect-snapped` gate, as it must. - `bench/runs/bfp-quant-synthetic-20260924T094254Z.jsonl`: **synthetic only.** GPTQ beats RTN on `tr(E H Eᵀ)/tr(W H Wᵀ)` in every synthetic layer, by 16-23x. - That size of gain comes from a steep synthetic spectrum with outlier channels. It is **not a prediction for Qwen**. - Unit test: GPTQ with H = I reduces exactly to RTN. - `bench/runs/bfp-proxy-hfree-20260924T092920Z.jsonl`: Qwen3.8-27B-FP8, layers 0-3, 30, 31, 62, 63 plus embed and lm_head. **Hessian-free proxy (H = I)**, measured against the FP8 weights dequantised the way prod does. - RTN bfp4 relMSE is 1.31-1.47e-2 in every group. The lowest groups are MLP gate/up/down; the highest are GDN out_proj and attention v_proj. - RTN bfp8 is 5.5-6.4e-5, about 235x lower. - 8-10% of bfp4 blocks saturate at the block max. - The {e, e+1} exponent search gains at most 0.1%, so it is not worth having. - The groups differ very little because bfp4 has roughly constant relative precision. **Ranking groups for P3 needs Hessians and P1, not weight statistics.** This proxy also cannot rank LDLQ, since GPTQ with H = I is RTN. ### Compute findings and blockers - **No suitable calibration compute is known.** - cfx-llm0 has 4x RTX 3080 10 GB. It is production vLLM serving 27B at TP=4, so using it means taking cluster inference down, which needs owner sign-off. - `nixos` (and `orchestra`, a container on it) has 47 GB RAM, 20 cores and an AMD 6900 XT (ROCm, no CUDA). It is the build/gaming host. - cfx-llm2 has 15 GB RAM and 4 cores. cfx-llm4 is Vulkan AMD. cfx-mgmt has a 6 GB 2060. - No host is marked for batch ML work. - Options: rent an A100/H100 or a CUDA box with at least 128 GB RAM; get approval for a cfx-llm0 window; or run a slow layer-streamed CPU calibration on `nixos`, which would load the shared host. - **No bf16 checkpoint anywhere.** Only `Qwen/Qwen3.8-27B-FP8` exists, on cfx-llm2 at `/var/lib/models/hf/hub/models--Qwen--Qwen3.8-27B-FP8`, snapshot `017b9c7a`. So bf4-from-bf16 vs bf4-from-FP8 is **not measured**. `orchestra` has 20 GB of free disk and cannot stage 54 GB. - **Target checkpoint change (owner):** calibration and error studies should target **bottlecapai/ThinkingCap-Qwen3.8-27B** bf16 (FP8 variant: `-FP8`) once gated access is granted. Nothing in the tool assumes Qwen paths or the name of the bf16-kept-modules key. - **EXL3 comparison:** exl3xpu `ref.py` (MIT) only reconstructs tensors from an existing EXL3 checkpoint. It needs an EXL3 4.0/3.0 bpw checkpoint of the same base, plus the bf16 reference, to be meaningful, so it is not done yet. ### Next step 1. Get GPU access or approval, and bf16 (ThinkingCap) access. 2. Run exllamav3-style Hessian collection (sc_trace-style self-sampled calibration), saving one `H` per module as `<module>.safetensors`, which `quantize --hessians` already reads. 3. Compute proxy arms (i)-(iii) per group, check them against the go/no-go criteria, then add (v) EXL3 via `ref.py`. P0 checkbox is left unticked: the emulation and quantizer parts are done, but the Hessian-weighted study is blocked on compute.
Grok commented 2026-09-24 21:25:54 +02:00
Author
Owner
Copy link

P0 result: calibration-Hessian proxy study on ThinkingCap (branch bfp-quant, 90ac45b)

Calibration (MEASURED, cfx-llm0, 4x RTX 3080, owner-approved window).

  • Source: ThinkingCap-Qwen3.8-27B bf16, snapshot 52361b72.
  • Data: 256 x 2048 tokens rendered with the ThinkingCap chat template:
    • 108 codeforces-cots and 96 OpenR1-Math reasoning traces (with <think>), both from Mixture-of-Thoughts.
    • 52 packed hermes function-calling windows.
    • tokens sha256 969719773e92fc28.
  • Method: layer by layer, using the HF Qwen3_5DecoderLayer reference. 21.5 min for all 64 layers.
  • Output: Hessians at cfx-llm0:/var/lib/models/calib/H-thinkingcap-bf16 (118 GB, with manifest.json).
  • Forward sanity check: CE 3.748 on 255 tokens, vs 3.677 from production vLLM on the same tokens.
  • Gap: the lm_head Hessian was lost to a file-name collision. The bug is fixed but lm_head was not re-collected.

Downtime. vllm-pod was down 2026-09-24 10:52:24Z - 11:17:30Z (~25 min).

  • A PATH bug in the first launch caused a 3 s stop/start at 10:52:24Z.
  • The real window ran from 10:53:26Z (stop) to 11:15:21Z (restart issued).
  • Serving verified at 11:17:30Z: container up, GPUs at 8.5 GB each, a real chat completion returned tokens.
  • The study then ran on orchestra's CPU, streaming one layer at a time, with no further downtime.

Study (MEASURED proxy, bench/runs/bfp-quant-hessian-proxy-20260924T192517Z.jsonl).

  • Proxy: tr(E H Eᵀ)/tr(W H Wᵀ), with W the bf16 source and H the calibration Hessian.
  • Coverage: 25 of 64 layers, 184 modules, every output checked on-grid.
  • Held-out layers: GPTQ is fit on calibration half A and scored on half B.
group RTN bf4 (prod packer, bf16 src) RTN bf4 from FP8 src GPTQ bf4 gain RTN bf8 FP8 ckpt itself
attn_qkv (48) 4.80e-3 5.11e-3 7.6e-4 5.68x held-out 2.07e-5 3.0e-4
attn_o (16) 8.68e-3 9.10e-3 1.83e-3 3.88x held-out 3.84e-5 4.2e-4
gdn_qkvz (18) 4.99e-3 5.20e-3 8.5e-4 5.90x in-sample 2.19e-5 2.1e-4
gdn_ab (18) 5.39e-3 5.39e-3 4.3e-4 12.5x in-sample 2.33e-5 0 (unquantized in FP8)
gdn_out (9) 1.11e-2 1.15e-2 4.6e-3 2.40x in-sample 5.51e-5 5.1e-4
mlp_gate_up (50) 4.66e-3 4.90e-3 1.60e-3 2.63x held-out 2.02e-5 2.4e-4
mlp_down (25) 6.53e-3 6.83e-3 1.56e-3 3.71x held-out 3.12e-5 2.8e-4

Go/no-go.

  • First condition (≥2x on attn/GDN): met on held-out data for attention. GDN is in-sample only, because every held-out layer is a full-attention layer.
  • The worst measured held-out/in-sample shrink is 0.48 (median 0.89). Applied to GDN as an ESTIMATE, it leaves gdn_out at 1.14x. That is the weakest group.
  • Second condition (within 2x of EXL3-4bpw): NOT MEASURED. There is no EXL3 build of this base on the cluster.
  • Verdict: GO-PENDING-EXL3.

Findings that matter for the next step.

  • GPTQ bf4 on attention is still ~36x the proxy error of RTN bf8, and ~2.5x the FP8 checkpoint's own error. Moving attn/GDN from bf8 to bf4 is a real quality cost, and only the device gate can price it.
  • Packing bf8 from the FP8 checkpoint carries ~15x more error than packing it from bf16 (3.2e-4 vs 2.1e-5). The FP8 source error dominates, so a bf16 source is a free quality win for the bf8 tensors.

Next (starting now):

  • A GPTQ-snapped ThinkingCap bfp4 checkpoint, in two arms: MLP at the same bytes, then attn/GDN moved to bf4.
  • A 100% bitwise real-packer round-trip check on it.
  • The P1 on-card accuracy gate: KL/top-1 vs bf16 at deep positions, retrieval, GSM8K with thinking on. It will compare prod RTN against the GPTQ arms, each with its own tensor cache dir, plus the standard speed sweep.

Runbook: .claude/skills/bfp-quant/SKILL.md.

## P0 result: calibration-Hessian proxy study on ThinkingCap (branch `bfp-quant`, 90ac45b) **Calibration (MEASURED, cfx-llm0, 4x RTX 3080, owner-approved window).** - Source: ThinkingCap-Qwen3.8-27B bf16, snapshot 52361b72. - Data: 256 x 2048 tokens rendered with the ThinkingCap chat template: - 108 codeforces-cots and 96 OpenR1-Math reasoning traces (with `<think>`), both from Mixture-of-Thoughts. - 52 packed hermes function-calling windows. - tokens sha256 969719773e92fc28. - Method: layer by layer, using the HF `Qwen3_5DecoderLayer` reference. 21.5 min for all 64 layers. - Output: Hessians at `cfx-llm0:/var/lib/models/calib/H-thinkingcap-bf16` (118 GB, with `manifest.json`). - Forward sanity check: CE 3.748 on 255 tokens, vs 3.677 from production vLLM on the same tokens. - Gap: the lm_head Hessian was lost to a file-name collision. The bug is fixed but lm_head was not re-collected. **Downtime.** vllm-pod was down 2026-09-24 **10:52:24Z - 11:17:30Z (~25 min)**. - A PATH bug in the first launch caused a 3 s stop/start at 10:52:24Z. - The real window ran from 10:53:26Z (stop) to 11:15:21Z (restart issued). - Serving verified at 11:17:30Z: container up, GPUs at 8.5 GB each, a real chat completion returned tokens. - The study then ran on orchestra's CPU, streaming one layer at a time, with **no further downtime**. **Study (MEASURED proxy, `bench/runs/bfp-quant-hessian-proxy-20260924T192517Z.jsonl`).** - Proxy: tr(E H Eᵀ)/tr(W H Wᵀ), with W the bf16 source and H the calibration Hessian. - Coverage: 25 of 64 layers, 184 modules, every output checked on-grid. - Held-out layers: GPTQ is fit on calibration half A and scored on half B. | group | RTN bf4 (prod packer, bf16 src) | RTN bf4 from FP8 src | GPTQ bf4 | gain | RTN bf8 | FP8 ckpt itself | |---|---|---|---|---|---|---| | attn_qkv (48) | 4.80e-3 | 5.11e-3 | 7.6e-4 | **5.68x** held-out | 2.07e-5 | 3.0e-4 | | attn_o (16) | 8.68e-3 | 9.10e-3 | 1.83e-3 | **3.88x** held-out | 3.84e-5 | 4.2e-4 | | gdn_qkvz (18) | 4.99e-3 | 5.20e-3 | 8.5e-4 | 5.90x in-sample | 2.19e-5 | 2.1e-4 | | gdn_ab (18) | 5.39e-3 | 5.39e-3 | 4.3e-4 | 12.5x in-sample | 2.33e-5 | 0 (unquantized in FP8) | | gdn_out (9) | 1.11e-2 | 1.15e-2 | 4.6e-3 | 2.40x in-sample | 5.51e-5 | 5.1e-4 | | mlp_gate_up (50) | 4.66e-3 | 4.90e-3 | 1.60e-3 | **2.63x** held-out | 2.02e-5 | 2.4e-4 | | mlp_down (25) | 6.53e-3 | 6.83e-3 | 1.56e-3 | **3.71x** held-out | 3.12e-5 | 2.8e-4 | **Go/no-go.** - First condition (≥2x on attn/GDN): **met** on held-out data for attention. GDN is in-sample only, because every held-out layer is a full-attention layer. - The worst measured held-out/in-sample shrink is 0.48 (median 0.89). Applied to GDN as an ESTIMATE, it leaves gdn_out at 1.14x. That is the weakest group. - Second condition (within 2x of EXL3-4bpw): **NOT MEASURED**. There is no EXL3 build of this base on the cluster. - Verdict: **GO-PENDING-EXL3**. **Findings that matter for the next step.** - GPTQ bf4 on attention is still ~36x the proxy error of RTN bf8, and ~2.5x the FP8 checkpoint's own error. Moving attn/GDN from bf8 to bf4 is a real quality cost, and only the device gate can price it. - Packing bf8 from the FP8 checkpoint carries ~15x more error than packing it from bf16 (3.2e-4 vs 2.1e-5). The FP8 source error dominates, so a bf16 source is a free quality win for the bf8 tensors. **Next (starting now):** - A GPTQ-snapped ThinkingCap bfp4 checkpoint, in two arms: MLP at the same bytes, then attn/GDN moved to bf4. - A 100% bitwise real-packer round-trip check on it. - The P1 on-card accuracy gate: KL/top-1 vs bf16 at deep positions, retrieval, GSM8K with thinking on. It will compare prod RTN against the GPTQ arms, each with its own tensor cache dir, plus the standard speed sweep. Runbook: `.claude/skills/bfp-quant/SKILL.md`.
Grok commented 2026-09-24 21:56:59 +02:00
Author
Owner
Copy link

Gate wording corrected (owner, 2026-09-24): the FP8 checkpoint is the reference, not bf16

The reference is now the native ThinkingCap-Qwen3.8-27B-FP8 checkpoint (e4m3, 128x128 blocks), dequantised the way production does it. We treat it as lossless.

What changes:

  • Gates: every gate is now a delta against FP8.
    • Proxy error per group is tr(E H Eᵀ)/tr(W8 H W8ᵀ), with E = Q − W8. The Hessians from the bf16 calibration run stay valid, because they are activation statistics.
    • The P1 teacher is FP8 served natively by vLLM on cfx-llm0 (Marlin FP8, bf16 activations and KV). P1 reports KL and top-1 agreement against that teacher.
    • GSM8K (thinking on, with completion length) and keyed retrieval are reported as deltas vs FP8.
    • bf16 appears only as a secondary sanity row.
  • The bf16-referenced P0 record (bench/runs/bfp-quant-hessian-proxy-20260924T192517Z.jsonl) is kept, but it is superseded as the headline. It is re-run against W8.
  • The quantizer source is now FP8.
    • tools/bfp-quant/calib/quantize_ckpt.py runs GPTQ from W8 (dequant_fp8_block, exactly as prod does).
    • The bf16 source is measured only as a labelled comparison: on held-out layers 3/31/63, both are scored vs W8.
    • First data point, layer 3 k/v held-out: GPTQ-from-FP8 scores 1.32e-3 / 2.37e-3 vs W8. GPTQ-from-bf16 scores 1.88e-3 / 2.86e-3. So FP8 is the better source, as expected.

P1 arms (each with its own TT tensor cache dir; same prod-argmax serve and sources as tc-baseline):

arm MLP attn/GDN source
P RTN bf4 (prod) RTN bf8 (prod) FP8 checkpoint as served today
G1 GPTQ bf4, same bytes RTN bf8, identical to P FP8
R2 RTN bf4 RTN bf4 FP8 (attn/GDN moved to bf4, weights only, QWEN36_ATTN_GDN_BF4=1)
G2 GPTQ bf4 GPTQ bf4 FP8
  • Arm checkpoints are the FP8 base with the snapped modules swapped in as bf16 tensors, using quantization_config.ignore. Prod's lazy FP8 loader passes bf16 keys through untouched, so the model code does not change.
  • Quant units follow the prod device layout. in_proj_a and in_proj_b are fused per device as [a_d|b_d]; 24 rows each means one bfp block straddles them, so they are quantised jointly.
  • Round-trip gate: tools/bfp-quant/calib/verify_arm.py packs every per-device matrix through the real tt-metal packer. It passes only if all of them are bitwise fixed points.

Battery per arm (bench/bfpq/tt-arm.sh, detached, tt-card-slot lock, waits and does not preempt):

  • 44 deep-context probes: 4 long texts × depths 1k–32k, 32 greedy tokens each, top-20 logprobs. The FP8 teacher then scores each arm's own continuation.
  • kv-accuracy at 16k/32k.
  • GSM8K n=60, thinking on, 8192 max tokens.
  • Standard sweep 4k/8k/16k/32k × B{4,8} × 1024 out, reporting prefill tok/s, TG ms/token and makespan.

Downtime so far today:

  • cfx-llm0 was down 19:39:11Z–19:43:09Z (~4 min) for a GPU GPTQ attempt. It hit a 10 GB OOM in RTN; that is fixed by chunking.
  • GPTQ now runs on CPU (orchestra, plus nice'd CPU on cfx-llm0) with no downtime.
  • The FP8-teacher window comes at the end: one window of roughly 45 min, covering the teacher scoring, GSM8K and retrieval.
## Gate wording corrected (owner, 2026-09-24): the FP8 checkpoint is the reference, not bf16 The reference is now the native ThinkingCap-Qwen3.8-27B-FP8 checkpoint (e4m3, 128x128 blocks), dequantised the way production does it. We treat it as lossless. **What changes:** - **Gates:** every gate is now a delta against FP8. - Proxy error per group is tr(E H Eᵀ)/tr(W8 H W8ᵀ), with E = Q − W8. The Hessians from the bf16 calibration run stay valid, because they are activation statistics. - The P1 teacher is FP8 served natively by vLLM on cfx-llm0 (Marlin FP8, bf16 activations and KV). P1 reports KL and top-1 agreement against that teacher. - GSM8K (thinking on, with completion length) and keyed retrieval are reported as deltas vs FP8. - bf16 appears only as a secondary sanity row. - **The bf16-referenced P0 record** (`bench/runs/bfp-quant-hessian-proxy-20260924T192517Z.jsonl`) is kept, but it is superseded as the headline. It is re-run against W8. - **The quantizer source is now FP8.** - `tools/bfp-quant/calib/quantize_ckpt.py` runs GPTQ from W8 (`dequant_fp8_block`, exactly as prod does). - The bf16 source is measured only as a labelled comparison: on held-out layers 3/31/63, both are scored vs W8. - First data point, layer 3 k/v held-out: GPTQ-from-FP8 scores 1.32e-3 / 2.37e-3 vs W8. GPTQ-from-bf16 scores 1.88e-3 / 2.86e-3. So FP8 is the better source, as expected. **P1 arms** (each with its own TT tensor cache dir; same prod-argmax serve and sources as tc-baseline): | arm | MLP | attn/GDN | source | |---|---|---|---| | P | RTN bf4 (prod) | RTN bf8 (prod) | FP8 checkpoint as served today | | G1 | GPTQ bf4, same bytes | RTN bf8, identical to P | FP8 | | R2 | RTN bf4 | RTN bf4 | FP8 (attn/GDN moved to bf4, weights only, `QWEN36_ATTN_GDN_BF4=1`) | | G2 | GPTQ bf4 | GPTQ bf4 | FP8 | - Arm checkpoints are the FP8 base with the snapped modules swapped in as bf16 tensors, using `quantization_config.ignore`. Prod's lazy FP8 loader passes bf16 keys through untouched, so the model code does not change. - Quant units follow the prod device layout. in_proj_a and in_proj_b are fused per device as [a_d|b_d]; 24 rows each means one bfp block straddles them, so they are quantised jointly. - **Round-trip gate:** `tools/bfp-quant/calib/verify_arm.py` packs every per-device matrix through the real tt-metal packer. It passes only if all of them are bitwise fixed points. **Battery per arm** (`bench/bfpq/tt-arm.sh`, detached, tt-card-slot lock, waits and does not preempt): - 44 deep-context probes: 4 long texts × depths 1k–32k, 32 greedy tokens each, top-20 logprobs. The FP8 teacher then scores each arm's own continuation. - kv-accuracy at 16k/32k. - GSM8K n=60, thinking on, 8192 max tokens. - Standard sweep 4k/8k/16k/32k × B{4,8} × 1024 out, reporting prefill tok/s, TG ms/token and makespan. **Downtime so far today:** - cfx-llm0 was down 19:39:11Z–19:43:09Z (~4 min) for a GPU GPTQ attempt. It hit a 10 GB OOM in RTN; that is fixed by chunking. - GPTQ now runs on CPU (orchestra, plus nice'd CPU on cfx-llm0) with no downtime. - The FP8-teacher window comes at the end: one window of roughly 45 min, covering the teacher scoring, GSM8K and retrieval.
Grok commented 2026-09-25 02:39:13 +02:00
Author
Owner
Copy link

P1 status: GPTQ-from-FP8 bfp4_b checkpoint built, verified and on-card tested; FP8 teacher gate still PENDING

Branch bfp-quant @ e5e29ec. All numbers below are MEASURED unless marked otherwise.

1. Checkpoint and verification

  • GPTQ/LDLQ was run from dequantized FP8 (W8), the source of truth, onto the tt-metal bfp4_b grid. It covers all 64 layers and every attn/GDN/MLP linear, in the prod per-device layout (tp=2; a|b jointly blocked). Runtime: one cfx-llm0 window, 4x3080.
  • Real-packer round-trip: 64/64 layers, 24.36e9 elements, 0 mismatches, 0 non-bf16 values. A fault-injection control (+1 ulp on every 97th value) must mismatch, and does (min 3,379 per layer). Record: bench/runs/bfp-quant-arm-roundtrip-20260925T0030Z.jsonl.

2. Proxy error vs FP8

Relative output error tr(dW H dWᵀ)/tr(W H Wᵀ), where W = dequantized FP8. Record: bench/runs/bfp-quant-hessian-proxy-fp8ref-20260925T0030Z.jsonl.

In-sample, all 64 layers (unit means):

group RTN bf4 GPTQ bf4 gain
attn_qkv 8.50e-3 1.70e-3 5.0x
attn_o 1.32e-2 3.30e-3 4.0x
gdn_qkvz 8.70e-3 1.29e-3 6.8x
gdn_ab 8.65e-3 5.15e-4 16.8x
gdn_out 1.05e-2 4.74e-3 2.2x
mlp_gate_up 8.52e-3 3.34e-3 2.6x
mlp_down 1.01e-2 4.73e-3 2.1x

Held-out, layers 3/31/63: GPTQ is fitted on half the calibration sequences and scored on the other half.

group GPTQ bf4 RTN bf8 gain vs RTN bf4
attn_qkv 1.51e-3 3.4e-5 5.1x
attn_o 2.58e-3 4.5e-5 4.1x
mlp_gate_up 2.74e-3 2.9e-5 2.5x
mlp_down 4.57e-3 3.5e-5 1.8x

Notes on the held-out table:

  • In-sample figures overstate the gain; mlp_down drops from 2.1x in-sample to 1.8x held-out. Held-out coverage has no GDN units, because layers 3/31/63 are full-attention layers.
  • GPTQ bf4 is still ~40-130x farther from FP8 than RTN bf8.
  • Quantize from FP8, not bf16: GPTQ from bf16, scored vs FP8, is worse in 21/21 held-out units.

3. On cards (cfx-llm2, prod-argmax, CTX 34816, SEQS 8, bf8 KV + device argmax)

Each arm has its own tensor cache. Every arm shows the same prod banners. The attn/GDN bf4 arms show QWEN36_ATTN_GDN_BF4 engaged x64; the MLP-only arms show it x0.

arm weights GSM8K n=60 (think) median completion tok kv-acc 16k/32k
P (prod) RTN: MLP bf4, attn/GDN bf8 57/60 145 1.0 / 1.0
G1 GPTQ MLP bf4, attn/GDN bf8 58/60 156.5 1.0 / 1.0
R2 RTN: all bf4 56/60 159.5 1.0 / 1.0
G2 GPTQ: all bf4 58/60 161 1.0 / 1.0

Speed (n=1 per cell). TG = decode ms/token. Makespan is for 1024 output tokens.

B ctx P TG G2 TG P makespan G2 makespan prefill tok/s P / G2
4 4k 52.48 49.49 59.81 s 56.72 s 2733 / 2736
4 32k 56.67 54.27 97.12 95.11 3148 / 3182
8 4k 52.87 51.04 70.64 67.10 2725 / 2798
8 32k 63.02 61.17 146.46 143.77 3140 / 3180

Full 16-cell table for all four arms: bench/runs/bfp-quant-p1-20260925T0045Z.jsonl. Per-arm raw data: bench/runs/bfpq-p1-arms/.

What the on-card results show:

  • attn/GDN at bf4 is worth 2-4 ms/token decode and 1.4-6% makespan. The gain is the smaller weight read.
  • Prefill is unchanged (compute-bound).
  • GPTQ vs RTN at the same format costs nothing at runtime; G2 lands within ~1 ms of R2.
  • Task accuracy cannot separate the arms. GSM8K deltas are 1-2 questions (1 question = 1.7 pp), and kv-accuracy is at ceiling for every arm.
  • Continuations are identical to P's on only 17-20 of 44 deep probes for every non-prod arm, R2 included. The arms clearly diverge, but that measure has no direction: whether GPTQ moves them toward FP8 is exactly what the FP8 teacher KL is for.

4. FP8 teacher (KL/top-1 at deep positions, plus FP8 GSM8K/kv rows): PENDING

Three short cfx-llm0 windows failed, each on an environment difference between the job container and prod:

  1. 2026-09-24 23:22:41Z–23:41:11Z (vllm-pod healthy again at 23:41:11Z). flashinfer's sampler JIT-builds and needs ninja, which the image lacks; prod hides this behind its JIT cache.
  2. 2026-09-25 00:23:25Z–00:29:23Z. The vision-encoder profile left -0.46 GiB for KV.
  3. 2026-09-25 00:29:33Z–00:35:25Z. The engine came up, but the fp32 prompt_logprobs tensor (486 MiB per 512-token chunk) OOMed; vLLM's memory profile does not count it.

Each restore was health-checked with a real completion. The fix is in e5e29ec:

  • chunk 256, util 0.88
  • no vision encoder
  • identical continuations deduplicated (2.50M → 1.68M tokens)
  • resumable, deadline-bounded scoring
  • dry-run against a fake engine, including a crash and resume

The next run needs one continuous ≤45-min window.

Provisional go/no-go vs the FP8-referenced gates

  • Build/verify gate: PASS. Bit-exact real-packer round-trip, non-blind.
  • Proxy gate: PASS for GPTQ over RTN. Held-out gain is 1.8-5.1x vs FP8.
  • Task gate: no regression detected vs prod (G1/G2 58/60 vs P 57/60, kv at ceiling). Not yet expressed as a delta vs FP8.
  • KL/top-1-vs-FP8 gate: PENDING. It is the only gate that can rank these arms, so G2 (all-bf4 GPTQ) is a candidate, not a GO, until it lands.
## P1 status: GPTQ-from-FP8 bfp4_b checkpoint built, verified and on-card tested; FP8 teacher gate still PENDING Branch `bfp-quant` @ e5e29ec. All numbers below are MEASURED unless marked otherwise. ### 1. Checkpoint and verification - GPTQ/LDLQ was run from **dequantized FP8 (W8)**, the source of truth, onto the tt-metal bfp4_b grid. It covers all 64 layers and every attn/GDN/MLP linear, in the prod per-device layout (tp=2; a|b jointly blocked). Runtime: one cfx-llm0 window, 4x3080. - **Real-packer round-trip: 64/64 layers, 24.36e9 elements, 0 mismatches, 0 non-bf16 values.** A fault-injection control (+1 ulp on every 97th value) must mismatch, and does (min 3,379 per layer). Record: `bench/runs/bfp-quant-arm-roundtrip-20260925T0030Z.jsonl`. ### 2. Proxy error vs FP8 Relative output error tr(dW H dWᵀ)/tr(W H Wᵀ), where W = dequantized FP8. Record: `bench/runs/bfp-quant-hessian-proxy-fp8ref-20260925T0030Z.jsonl`. In-sample, all 64 layers (unit means): | group | RTN bf4 | GPTQ bf4 | gain | |---|---|---|---| | attn_qkv | 8.50e-3 | 1.70e-3 | 5.0x | | attn_o | 1.32e-2 | 3.30e-3 | 4.0x | | gdn_qkvz | 8.70e-3 | 1.29e-3 | 6.8x | | gdn_ab | 8.65e-3 | 5.15e-4 | 16.8x | | gdn_out | 1.05e-2 | 4.74e-3 | 2.2x | | mlp_gate_up | 8.52e-3 | 3.34e-3 | 2.6x | | mlp_down | 1.01e-2 | 4.73e-3 | 2.1x | Held-out, layers 3/31/63: GPTQ is fitted on half the calibration sequences and scored on the other half. | group | GPTQ bf4 | RTN bf8 | gain vs RTN bf4 | |---|---|---|---| | attn_qkv | 1.51e-3 | 3.4e-5 | 5.1x | | attn_o | 2.58e-3 | 4.5e-5 | 4.1x | | mlp_gate_up | 2.74e-3 | 2.9e-5 | 2.5x | | mlp_down | 4.57e-3 | 3.5e-5 | 1.8x | Notes on the held-out table: - In-sample figures overstate the gain; mlp_down drops from 2.1x in-sample to 1.8x held-out. Held-out coverage has no GDN units, because layers 3/31/63 are full-attention layers. - **GPTQ bf4 is still ~40-130x farther from FP8 than RTN bf8.** - **Quantize from FP8, not bf16:** GPTQ from bf16, scored vs FP8, is worse in **21/21** held-out units. ### 3. On cards (cfx-llm2, prod-argmax, CTX 34816, SEQS 8, bf8 KV + device argmax) Each arm has its own tensor cache. Every arm shows the same prod banners. The attn/GDN bf4 arms show `QWEN36_ATTN_GDN_BF4 engaged` x64; the MLP-only arms show it x0. | arm | weights | GSM8K n=60 (think) | median completion tok | kv-acc 16k/32k | |---|---|---|---|---| | P (prod) | RTN: MLP bf4, attn/GDN bf8 | 57/60 | 145 | 1.0 / 1.0 | | G1 | GPTQ MLP bf4, attn/GDN bf8 | 58/60 | 156.5 | 1.0 / 1.0 | | R2 | RTN: all bf4 | 56/60 | 159.5 | 1.0 / 1.0 | | G2 | GPTQ: all bf4 | 58/60 | 161 | 1.0 / 1.0 | Speed (n=1 per cell). TG = decode ms/token. Makespan is for 1024 output tokens. | B | ctx | P TG | G2 TG | P makespan | G2 makespan | prefill tok/s P / G2 | |---|---|---|---|---|---|---| | 4 | 4k | 52.48 | 49.49 | 59.81 s | 56.72 s | 2733 / 2736 | | 4 | 32k | 56.67 | 54.27 | 97.12 | 95.11 | 3148 / 3182 | | 8 | 4k | 52.87 | 51.04 | 70.64 | 67.10 | 2725 / 2798 | | 8 | 32k | 63.02 | 61.17 | 146.46 | 143.77 | 3140 / 3180 | Full 16-cell table for all four arms: `bench/runs/bfp-quant-p1-20260925T0045Z.jsonl`. Per-arm raw data: `bench/runs/bfpq-p1-arms/`. What the on-card results show: - **attn/GDN at bf4 is worth 2-4 ms/token decode and 1.4-6% makespan.** The gain is the smaller weight read. - **Prefill is unchanged** (compute-bound). - GPTQ vs RTN at the same format costs nothing at runtime; G2 lands within ~1 ms of R2. - **Task accuracy cannot separate the arms.** GSM8K deltas are 1-2 questions (1 question = 1.7 pp), and kv-accuracy is at ceiling for every arm. - Continuations are identical to P's on only 17-20 of 44 deep probes for every non-prod arm, R2 included. The arms clearly diverge, but that measure has no direction: whether GPTQ moves them *toward* FP8 is exactly what the FP8 teacher KL is for. ### 4. FP8 teacher (KL/top-1 at deep positions, plus FP8 GSM8K/kv rows): PENDING Three short cfx-llm0 windows failed, each on an environment difference between the job container and prod: 1. 2026-09-24 23:22:41Z–23:41:11Z (vllm-pod healthy again at 23:41:11Z). flashinfer's sampler JIT-builds and needs `ninja`, which the image lacks; prod hides this behind its JIT cache. 2. 2026-09-25 00:23:25Z–00:29:23Z. The vision-encoder profile left -0.46 GiB for KV. 3. 2026-09-25 00:29:33Z–00:35:25Z. The engine came up, but the fp32 prompt_logprobs tensor (486 MiB per 512-token chunk) OOMed; vLLM's memory profile does not count it. Each restore was health-checked with a real completion. The fix is in e5e29ec: - chunk 256, util 0.88 - no vision encoder - identical continuations deduplicated (2.50M → 1.68M tokens) - resumable, deadline-bounded scoring - dry-run against a fake engine, including a crash and resume The next run needs **one continuous ≤45-min window**. ### Provisional go/no-go vs the FP8-referenced gates - **Build/verify gate: PASS.** Bit-exact real-packer round-trip, non-blind. - **Proxy gate: PASS for GPTQ over RTN.** Held-out gain is 1.8-5.1x vs FP8. - **Task gate: no regression detected vs prod** (G1/G2 58/60 vs P 57/60, kv at ceiling). Not yet expressed as a delta vs FP8. - **KL/top-1-vs-FP8 gate: PENDING.** It is the only gate that can rank these arms, so **G2 (all-bf4 GPTQ) is a candidate, not a GO**, until it lands.
Grok commented 2026-09-25 08:38:37 +02:00
Author
Owner
Copy link

P1 final: the FP8 teacher gate has landed. G2 (GPTQ-from-FP8, all bf4) is GO; R2 (RTN, all bf4) is NO-GO

Branch bfp-quant @ 2b1504d. Record: bench/runs/bfp-quant-p1-20260925T0645Z.jsonl. All numbers are MEASURED.

Teacher. Native FP8 ThinkingCap served with vLLM 0.27.1 on cfx-llm0: TP=4, bf16 KV, no prefix caching.

  • It ran in one continuous window: vllm-pod down 2026-09-25 05:09:25Z, healthy again 05:57:14Z (47.8 min), confirmed with a real completion.
  • Before that, three short failed windows (their causes are in the previous comment) gave about 30 min of extra downtime.

KL / top-1 vs FP8

  • Setup: 9 paired deep probes at 24k-32k depth, 32 steps each, with SE clustered by probe.
arm KL(arm‖FP8) Δ vs prod P top-1 = FP8 argmax Δ vs P −log p_FP8(arm token)
P (prod RTN, attn/GDN bf8) 0.111 ± 0.064 — 0.917 — 0.462
G1 (GPTQ MLP) 0.094 ± 0.051 −0.017 ± 0.014 0.906 −1.0 ± 4.0 pp 0.486
R2 (RTN, all bf4) 0.176 ± 0.094 +0.065 ± 0.045 0.920 +0.4 ± 2.1 pp 0.479
G2 (GPTQ, all bf4) 0.085 ± 0.037 −0.026 ± 0.039 0.944 +2.8 ± 1.4 pp 0.338

Task accuracy as a delta vs FP8

  • FP8 scores GSM8K 59/60 (thinking on) with median completion 147 tokens, and retrieval 1.0/1.0 at 16k/32k.
arm GSM8K Δ vs FP8 median completion (Δ) retrieval Δ
P 57/60 −2 q (−3.3 pp) 145 (−2) 0 / 0
G1 58/60 −1 q (−1.7 pp) 156.5 (+9.5) 0 / 0
R2 56/60 −3 q (−5.0 pp) 159.5 (+12.5) 0 / 0
G2 58/60 −1 q (−1.7 pp) 161 (+14) 0 / 0

Gates

The gate is non-inferiority against the deployed prod arm, with everything measured vs FP8.

  • G2: GO. It is the candidate for the default. It has the lowest KL to FP8, the highest top-1 and the lowest teacher NLL, is within 1 question of FP8 on GSM8K, and matches FP8 on retrieval.
    • Speed vs prod: decode is 1.8-3.3 ms/token faster, makespan is 1.8-5.2% shorter, and prefill is unchanged at ~2.7-3.2k tok/s.
    • Size: 4-bit attn/GDN, so fewer bytes than prod.
  • G1: GO. It is the same bytes and speed as prod, and accuracy is neutral to slightly better.
  • R2: NO-GO. It has the same bytes and speed as G2 but is the only arm trending worse than prod on KL. This is the core result: moving attn/GDN to bf4 costs accuracy under RTN, and GPTQ from FP8 buys it back.

Caveats

  • n = 9 probes. Only G2's top-1 gain reaches ~2 SE; the KL deltas are within 1-1.5 SE.
  • The teacher scored 32 of 115 unique continuations, deepest first, before its deadline. Shallow depths (≤16k) are unscored, but teacher_fp8.py is resumable.
  • KL is a lower bound over the arm's top-20.

Next: build a compact packed bfp4 checkpoint of G2 and publish it to the private HF repo (owner request).

## P1 final: the FP8 teacher gate has landed. G2 (GPTQ-from-FP8, all bf4) is GO; R2 (RTN, all bf4) is NO-GO Branch `bfp-quant` @ 2b1504d. Record: `bench/runs/bfp-quant-p1-20260925T0645Z.jsonl`. All numbers are MEASURED. **Teacher.** Native FP8 ThinkingCap served with vLLM 0.27.1 on cfx-llm0: TP=4, bf16 KV, no prefix caching. - It ran in **one continuous window: vllm-pod down 2026-09-25 05:09:25Z, healthy again 05:57:14Z (47.8 min)**, confirmed with a real completion. - Before that, three short failed windows (their causes are in the previous comment) gave about 30 min of extra downtime. ### KL / top-1 vs FP8 - Setup: 9 paired deep probes at 24k-32k depth, 32 steps each, with SE clustered by probe. | arm | KL(arm‖FP8) | Δ vs prod P | top-1 = FP8 argmax | Δ vs P | −log p_FP8(arm token) | |---|---|---|---|---|---| | P (prod RTN, attn/GDN bf8) | 0.111 ± 0.064 | — | 0.917 | — | 0.462 | | G1 (GPTQ MLP) | 0.094 ± 0.051 | −0.017 ± 0.014 | 0.906 | −1.0 ± 4.0 pp | 0.486 | | R2 (RTN, all bf4) | **0.176** ± 0.094 | **+0.065 ± 0.045** | 0.920 | +0.4 ± 2.1 pp | 0.479 | | G2 (GPTQ, all bf4) | **0.085** ± 0.037 | −0.026 ± 0.039 | **0.944** | **+2.8 ± 1.4 pp** | **0.338** | ### Task accuracy as a delta vs FP8 - FP8 scores GSM8K 59/60 (thinking on) with median completion 147 tokens, and retrieval 1.0/1.0 at 16k/32k. | arm | GSM8K | Δ vs FP8 | median completion (Δ) | retrieval Δ | |---|---|---|---|---| | P | 57/60 | −2 q (−3.3 pp) | 145 (−2) | 0 / 0 | | G1 | 58/60 | −1 q (−1.7 pp) | 156.5 (+9.5) | 0 / 0 | | R2 | 56/60 | −3 q (−5.0 pp) | 159.5 (+12.5) | 0 / 0 | | G2 | 58/60 | −1 q (−1.7 pp) | 161 (+14) | 0 / 0 | ### Gates The gate is non-inferiority against the deployed prod arm, with everything measured vs FP8. - **G2: GO.** It is the candidate for the default. It has the lowest KL to FP8, the highest top-1 and the lowest teacher NLL, is within 1 question of FP8 on GSM8K, and matches FP8 on retrieval. - Speed vs prod: decode is 1.8-3.3 ms/token faster, makespan is 1.8-5.2% shorter, and prefill is unchanged at ~2.7-3.2k tok/s. - Size: 4-bit attn/GDN, so fewer bytes than prod. - **G1: GO.** It is the same bytes and speed as prod, and accuracy is neutral to slightly better. - **R2: NO-GO.** It has the same bytes and speed as G2 but is the only arm trending *worse* than prod on KL. **This is the core result: moving attn/GDN to bf4 costs accuracy under RTN, and GPTQ from FP8 buys it back.** ### Caveats - n = 9 probes. Only G2's top-1 gain reaches ~2 SE; the KL deltas are within 1-1.5 SE. - The teacher scored 32 of 115 unique continuations, deepest first, before its deadline. Shallow depths (≤16k) are unscored, but `teacher_fp8.py` is resumable. - KL is a lower bound over the arm's top-20. **Next:** build a compact packed bfp4 checkpoint of G2 and publish it to the private HF repo (owner request).
Grok commented 2026-09-25 09:57:43 +02:00
Author
Owner
Copy link

G2 published as a compact bfp4 checkpoint (bfp4-packed-v1): 20.56 GB, private HF repo

Branch bfp-quant @ 65ebb7d. All numbers MEASURED.

Format. Each of the 496 snapped tensors is stored as its native tt-metal bfp4_b encoding:

  • 4-bit codes, two per byte (uint8 [in, out/2]), plus one exponent byte per 16-block (uint8 [in, out/16]), in the TT packer's own block order.
  • Shape and layout are recorded in the safetensors metadata.
  • The dead FP8 copies of those tensors are dropped: 496 weights + 400 weight_scale_inv = 896 keys.
  • Every other tensor (embed, lm_head, norms, conv1d, A_log/dt_bias, mtp.*, visual.*) is copied byte for byte.
  • quantization_config is removed, since no FP8 tensor is left.
  • Size: 20,559,210,616 bytes of safetensors (19.15 GiB), vs 31,239,067,400 for the FP8 base: −34%. The previous attempt was 68 GB.

Trap found by the packer's fail-closed check. GDN in_proj_a / in_proj_b are not on the grid as standalone tensors.

  • Prod fuses them per device ([a_d|b_d], 24 rows each), and GPTQ snapped them in that order, so a 16-block straddles a and b.
  • They are therefore stored as one joint entry per layer, ...in_proj_ab.weight, with its TP=2 split recorded. That is 448 packed entries for 496 tensors.

Tools.

  • tools/bfp-quant/bfp4_pack.py and bfp4_verify.py.
  • tools/bfp-quant/bfp4_unpack.py: library unpack / unpack_entry, plus the CLI bfp4_unpack.py PACKED_DIR OUT_DIR for non-TT users, which writes a plain bf16 checkpoint.
  • TT loader hook: bench/bfpq/sources/weight-mapping.py, the image's tt/weight_mapping.py (sha256 7e24c7dd…) plus the unpack path. It prints QWEN36_BFP4_PACKED engaged, and tt-arm.sh ARM_PACKED=1 fails closed without it.
  • 26 unit and toy end-to-end tests pass.

Gate 1, round-trip: PASS (bench/runs/bfp4-packed-roundtrip-20260925T072327Z.jsonl).

  • Unpack equals the ckpt-G2 snapped bf16 bit for bit for 496/496 tensors (24.35e9 elements).
  • 703 copied tensors are sha256-equal to the FP8 base.
  • The absent keys are exactly the 896 FP8 copies.
  • A fault-injection control (one flipped code bit) is caught.

Gate 2, TT boot from packed with a fresh cache: QUEUED, not run. The card lock has been held continuously by other agents (mtp-g2-ab, then tt-quant:gptq), and I did not preempt them.

  • bench/bfpq/packed-gate.sh is queued on cfx-llm2 under the lock protocol (log: /var/lib/models/bfpq/arm-PK.nohup).
  • When the slot frees it cold-boots the packed checkpoint (from /var/lib/models-sata, with a fresh cache). It passes only if every tensorbin it writes is sha256-identical to G2's cache (980 files, hashes already taken) and the 44 deep probes are token-identical to G2's.
  • The result will land in …/stage/PK/runs/*-PK/packed-gate.json.
  • Until then, the evidence is gate 1: bit-identical bf16 into ttnn.as_tensor.

Published: donach/ThinkingCap-Qwen3.8-27B-bfp4-GPTQ-tt. The HF API confirms it is private, with 22 files totalling 20,582,480,358 bytes and every size matching the local dir. Record: bench/runs/bfp4-packed-hf-publish-20260925T075700Z.jsonl.

  • Model card fixes:
    • It no longer claims the FP8 shards had the replaced tensors removed.
    • It describes the packed format and the unpack tool.
    • It carries the KL gate result (G2 GO) and keeps the private-candidate warning.
  • The PolyForm and Apache licence files and NOTICE are kept.
  • The token was piped over stdin into a 0600 file and removed on exit; it was confirmed gone.
  • Files: .gitattributes, LICENSE, LICENSE-Apache-2.0-Qwen.txt, LICENSE-PolyForm-Small-Business-1.0.0.txt, NOTICE, README.md, bfp4_packed.json, bfp4_unpack.py, chat_template.jinja, config.json, generation_config.json, merges.txt, model-0000{1..4}-of-00004.safetensors (7.14/5.59/4.44/3.39 GB), model.safetensors.index.json, preprocessor_config.json, tokenizer.json, tokenizer_config.json, video_preprocessor_config.json, vocab.json.

No new speed numbers. The packed checkpoint gives the device bit-identical weights, so G2's measured PP/TG apply once gate 2 confirms it.

## G2 published as a compact bfp4 checkpoint (`bfp4-packed-v1`): 20.56 GB, private HF repo Branch `bfp-quant` @ 65ebb7d. All numbers MEASURED. **Format.** Each of the 496 snapped tensors is stored as its native tt-metal bfp4_b encoding: - 4-bit codes, two per byte (uint8 `[in, out/2]`), plus one exponent byte per 16-block (uint8 `[in, out/16]`), in the TT packer's own block order. - Shape and layout are recorded in the safetensors metadata. - The dead FP8 copies of those tensors are dropped: 496 weights + 400 `weight_scale_inv` = 896 keys. - Every other tensor (embed, lm_head, norms, conv1d, A_log/dt_bias, `mtp.*`, `visual.*`) is copied byte for byte. - `quantization_config` is removed, since no FP8 tensor is left. - **Size: 20,559,210,616 bytes of safetensors (19.15 GiB), vs 31,239,067,400 for the FP8 base: −34%.** The previous attempt was 68 GB. **Trap found by the packer's fail-closed check.** GDN `in_proj_a` / `in_proj_b` are *not* on the grid as standalone tensors. - Prod fuses them per device (`[a_d|b_d]`, 24 rows each), and GPTQ snapped them in that order, so a 16-block straddles a and b. - They are therefore stored as one joint entry per layer, `...in_proj_ab.weight`, with its TP=2 split recorded. That is 448 packed entries for 496 tensors. **Tools.** - `tools/bfp-quant/bfp4_pack.py` and `bfp4_verify.py`. - `tools/bfp-quant/bfp4_unpack.py`: library `unpack` / `unpack_entry`, plus the CLI `bfp4_unpack.py PACKED_DIR OUT_DIR` for non-TT users, which writes a plain bf16 checkpoint. - TT loader hook: `bench/bfpq/sources/weight-mapping.py`, the image's `tt/weight_mapping.py` (sha256 7e24c7dd…) plus the unpack path. It prints `QWEN36_BFP4_PACKED engaged`, and `tt-arm.sh ARM_PACKED=1` fails closed without it. - 26 unit and toy end-to-end tests pass. **Gate 1, round-trip: PASS** (`bench/runs/bfp4-packed-roundtrip-20260925T072327Z.jsonl`). - Unpack equals the ckpt-G2 snapped bf16 **bit for bit for 496/496 tensors** (24.35e9 elements). - 703 copied tensors are sha256-equal to the FP8 base. - The absent keys are exactly the 896 FP8 copies. - A fault-injection control (one flipped code bit) is caught. **Gate 2, TT boot from packed with a fresh cache: QUEUED, not run.** The card lock has been held continuously by other agents (`mtp-g2-ab`, then `tt-quant:gptq`), and I did not preempt them. - `bench/bfpq/packed-gate.sh` is queued on cfx-llm2 under the lock protocol (log: `/var/lib/models/bfpq/arm-PK.nohup`). - When the slot frees it cold-boots the packed checkpoint (from `/var/lib/models-sata`, with a fresh cache). It passes only if every tensorbin it writes is sha256-identical to G2's cache (980 files, hashes already taken) **and** the 44 deep probes are token-identical to G2's. - The result will land in `…/stage/PK/runs/*-PK/packed-gate.json`. - Until then, the evidence is gate 1: bit-identical bf16 into `ttnn.as_tensor`. **Published:** `donach/ThinkingCap-Qwen3.8-27B-bfp4-GPTQ-tt`. The HF API confirms it is **private**, with 22 files totalling 20,582,480,358 bytes and every size matching the local dir. Record: `bench/runs/bfp4-packed-hf-publish-20260925T075700Z.jsonl`. - Model card fixes: - It no longer claims the FP8 shards had the replaced tensors removed. - It describes the packed format and the unpack tool. - It carries the KL gate result (G2 GO) and keeps the private-candidate warning. - The PolyForm and Apache licence files and NOTICE are kept. - The token was piped over stdin into a 0600 file and removed on exit; it was confirmed gone. - Files: `.gitattributes, LICENSE, LICENSE-Apache-2.0-Qwen.txt, LICENSE-PolyForm-Small-Business-1.0.0.txt, NOTICE, README.md, bfp4_packed.json, bfp4_unpack.py, chat_template.jinja, config.json, generation_config.json, merges.txt, model-0000{1..4}-of-00004.safetensors (7.14/5.59/4.44/3.39 GB), model.safetensors.index.json, preprocessor_config.json, tokenizer.json, tokenizer_config.json, video_preprocessor_config.json, vocab.json`. **No new speed numbers.** The packed checkpoint gives the device bit-identical weights, so G2's measured PP/TG apply once gate 2 confirms it.
Grok commented 2026-09-25 10:06:46 +02:00
Author
Owner
Copy link

"MLP bf4 retired" (old ledger 0aaa) and "attn/GDN stay 8-bit" are reopened. Neither was measured for accuracy, and the owner said "keep BF4" on 09-15. Your P1 result is now the record: G2 is non-inferior against the FP8 teacher (bfp-quant:bench/runs/bfp-quant-p1-20260925T0645Z.jsonl). lm_head bf4 is still untested.

Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).

"MLP bf4 retired" (old ledger 0aaa) and "attn/GDN stay 8-bit" are reopened. Neither was measured for accuracy, and the owner said "keep BF4" on 09-15. Your P1 result is now the record: G2 is non-inferior against the FP8 teacher (`bfp-quant:bench/runs/bfp-quant-p1-20260925T0645Z.jsonl`). lm_head bf4 is still untested. Reopened under the new rule (owner, 2026-09-25): **a closure needs an on-hardware record at the stated scope**; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).
Grok commented 2026-09-25 15:07:28 +02:00
Author
Owner
Copy link

Packed G2: on-card gate PASSED

bench/runs/bfp4-packed-tt-boot-gate-20260925T110520Z.jsonl (MEASURED, cfx-llm2, 2x P150a, TP=2, prod-argmax).

Setup: cold boot from the bfp4-packed-v1 checkpoint (20.56 GB), using a fresh TT_CACHE_PATH and the bfpq loader hook. The cold build took 1271 s.

Results:

  • QWEN36_BFP4_PACKED engaged, and QWEN36_ATTN_GDN_BF4 engaged x64.
  • The served root is the packed dir.
  • 980/980 tensorbins the build writes are sha256-identical to those built from the bf16-grid G2 checkpoint.
  • 44/44 deep probes are token-identical to G2's.

So the packed checkpoint is bit-identical on the device to the one that passed P1. All G2 accuracy and speed numbers carry over. There are no new speed numbers.

Earlier attempt: it died when another agent filled /var/lib/models-sata, after 849/849 identical files. That attempt is recorded as ...-try1.

This run: it completed at 11:05Z, before today's host reboot, so nothing needed re-queueing. Future bfpq TP=2 runs go through the ttx mesh slot (bench/bfpq/mesh-slot-run.sh).

HF: the model card on donach/ThinkingCap-Qwen3.8-27B-bfp4-GPTQ-tt now states this result. Re-verified via API: private, 22 files, sizes match.

Branch bfp-quant @ HEAD.

## Packed G2: on-card gate PASSED `bench/runs/bfp4-packed-tt-boot-gate-20260925T110520Z.jsonl` (MEASURED, cfx-llm2, 2x P150a, TP=2, prod-argmax). **Setup:** cold boot from the `bfp4-packed-v1` checkpoint (20.56 GB), using a **fresh** `TT_CACHE_PATH` and the bfpq loader hook. The cold build took 1271 s. **Results:** - `QWEN36_BFP4_PACKED engaged`, and `QWEN36_ATTN_GDN_BF4 engaged` x64. - The served root is the packed dir. - **980/980 tensorbins the build writes are sha256-identical** to those built from the bf16-grid G2 checkpoint. - **44/44 deep probes are token-identical** to G2's. **So the packed checkpoint is bit-identical on the device to the one that passed P1.** All G2 accuracy and speed numbers carry over. There are no new speed numbers. **Earlier attempt:** it died when another agent filled `/var/lib/models-sata`, after 849/849 identical files. That attempt is recorded as `...-try1`. **This run:** it completed at 11:05Z, before today's host reboot, so nothing needed re-queueing. Future bfpq TP=2 runs go through the ttx mesh slot (`bench/bfpq/mesh-slot-run.sh`). **HF:** the model card on `donach/ThinkingCap-Qwen3.8-27B-bfp4-GPTQ-tt` now states this result. Re-verified via API: private, 22 files, sizes match. Branch `bfp-quant` @ HEAD.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
mtp-thinkingcap
tt-quant
ttx
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#62
No description provided.