goal 4: Qwen3.8-27B on ONE Blackhole card, from the QUASAR NVFP4 checkpoint

multica-agent commented

2026-09-05 06:11:34 +02:00

Member

Docs and scaffolding for the new operator goal (2026-09-05): get a smaller quant running natively on a single card, using QUASAR NVFP4.

Stacked on #14 (goals 2-3). No behaviour change on any existing profile.

What is in here

docs/GOAL-NVFP4-SINGLE-CARD.md — the goal: what "natively" means on both axes, four gates (G4.1-G4.4), the measured budget, what fits, expected decode speed, the work in dependency order, and the ways it fails.
A goal4 profile in scripts/vllm-tt-serve.sh, committed non-functional on purpose — the unpacker and the coverage gate do not exist yet, but the mesh syntax needs recording.
Two corrections to standing documents.

The budget, measured

Live device weights sum to 25.853 GiB against 30.87 GiB usable per card — 5.02 GiB left, which will not serve long context. QUASAR's coverage takes the four bf8 groups it quantised down to bf4, saving 3.374 GiB → 22.479 GiB, freeing 8.39 GiB.

Against 34,816 B/token of KV (16 full-attention layers; the 48 GDN layers hold a fixed-size state), one 262,144 session needs 8.50 GiB — 1.1 GiB short until tok_embeddings also drops to bf8. That lever is held in reserve, not assumed. Goal 2 missed its first attempt at this margin by 0.17 GiB.

Corrections

QUASAR does not quantise all Linear. Its config ignores lm_head. All three sources — unsloth, QUASAR, and tt-metal's own policy — independently keep lm_head at 8 bits. Settled; should not cost an experiment.
FP4-PRIORITY.md's ~29 tok/s ceiling used 27.49 GiB, which counts prefill-only gate_up and the embedding table. The decode read is 20.17 GiB. More to the point, goal 3 showed the ceiling is not the binding constraint: ~80% of per-token time is fixed cost, and down_proj → bf4 bought +3.1% where that section predicted +15%. The prize here is footprint — which is exactly what makes one card possible — not speed.

Two things worth flagging

This goal incidentally runs the TP=1 diagnostic goal 3 asked for. If single-stream decode comes in much faster than the predicted 10-14 tok/s, the fixed cost was the TP=2 all-reduce, not the GDN chain.

The bf8 accuracy baseline is still uncaptured and is a prerequisite of G4.4. It has been outstanding since goal 3 and costs one two-card run.

Tier

T0 — documentation and a non-functional serve profile.

@hermes review please.

Docs and scaffolding for the new operator goal (2026-09-05): get a smaller quant running natively on a **single** card, using QUASAR NVFP4. Stacked on #14 (goals 2-3). No behaviour change on any existing profile. ## What is in here - `docs/GOAL-NVFP4-SINGLE-CARD.md` — the goal: what "natively" means on both axes, four gates (G4.1-G4.4), the measured budget, what fits, expected decode speed, the work in dependency order, and the ways it fails. - A `goal4` profile in `scripts/vllm-tt-serve.sh`, **committed non-functional on purpose** — the unpacker and the coverage gate do not exist yet, but the mesh syntax needs recording. - Two corrections to standing documents. ## The budget, measured Live device weights sum to **25.853 GiB** against **30.87 GiB usable** per card — 5.02 GiB left, which will not serve long context. QUASAR's coverage takes the four bf8 groups it quantised down to bf4, saving 3.374 GiB → **22.479 GiB**, freeing 8.39 GiB. Against 34,816 B/token of KV (16 full-attention layers; the 48 GDN layers hold a fixed-size state), one 262,144 session needs 8.50 GiB — **1.1 GiB short** until `tok_embeddings` also drops to bf8. That lever is held in reserve, not assumed. Goal 2 missed its first attempt at this margin by 0.17 GiB. ## Corrections - **QUASAR does not quantise all `Linear`.** Its config `ignore`s `lm_head`. All three sources — unsloth, QUASAR, and tt-metal's own policy — independently keep `lm_head` at 8 bits. Settled; should not cost an experiment. - **`FP4-PRIORITY.md`'s ~29 tok/s ceiling** used 27.49 GiB, which counts prefill-only `gate_up` and the embedding table. The decode read is 20.17 GiB. More to the point, goal 3 showed the ceiling is not the binding constraint: ~80% of per-token time is fixed cost, and `down_proj` → bf4 bought +3.1% where that section predicted +15%. **The prize here is footprint — which is exactly what makes one card possible — not speed.** ## Two things worth flagging **This goal incidentally runs the TP=1 diagnostic goal 3 asked for.** If single-stream decode comes in much faster than the predicted 10-14 tok/s, the fixed cost was the TP=2 all-reduce, not the GDN chain. **The bf8 accuracy baseline is still uncaptured** and is a prerequisite of G4.4. It has been outstanding since goal 3 and costs one two-card run. ## Tier T0 — documentation and a non-functional serve profile. @hermes review please.

multica-agent added 1 commit

2026-09-05 06:11:34 +02:00

goal 4: 27B on ONE card from QUASAR NVFP4 -- the goal, its budget, and two corrections 3c2819f273

Operator, 2026-09-05: get a smaller quant running natively on a single card,
using the QUASAR NVFP4 build.

"Natively" is a claim about two axes at once, and only stating both makes the
goal falsifiable. A 20.6 GB NVFP4 checkpoint loaded under today's dtype policy
gives byte-for-byte the same device footprint as the 30.9 GB FP8 one, because
tt-metal re-tilizes everything on load and picks its own dtype. So the gate is
(1) the compressed-tensors weights are unpacked by our loader, no side
conversion, and (2) the device carries bf4 where QUASAR carries 4-bit.

The budget is derived from measurement rather than the spec sheet, and it is
close. Live device weights today sum to 25.853 GiB against 30.87 GiB usable per
card -- 5.02 GiB left, which will not serve long context. QUASAR's coverage
takes the four bf8 groups it quantised down to bf4 and saves 3.374 GiB, landing
at 22.479 GiB and freeing 8.39 GiB. Against 34,816 B/token of KV -- 16 full
attention layers, the 48 GDN layers hold a fixed-size state -- one 262,144
session needs 8.50 GiB, so it is 1.1 GiB short until tok_embeddings also drops
to bf8. That is a real lever held in reserve, not an assumption; goal 2 missed
its first attempt at this margin by 0.17 GiB.

Two corrections to standing documents, both from reading the artefacts:

- QUASAR does NOT quantise all Linear. Its config ignores lm_head. All three
  sources -- unsloth, QUASAR, and tt-metal's own policy -- independently keep
  lm_head at 8 bits, so that question is settled and should not cost an
  experiment.
- FP4-PRIORITY's ~29 tok/s ceiling used the 27.49 GiB figure, which counts
  prefill-only gate_up and the embedding table. The decode read is 20.17 GiB.
  More importantly goal 3 showed the ceiling is not the binding constraint at
  all: ~80% of per-token time is fixed cost, and moving down_proj to bf4 bought
  +3.1% where that section predicted +15%. The prize here is footprint, which
  is exactly what makes one card possible -- it is not the speed prize.

Expect single-stream decode well under the 28.3 tok/s roofline, around 10-14,
and note that this goal incidentally runs the TP=1 diagnostic goal 3 asked for:
if it comes in much faster, the fixed cost was the all-reduce, not the GDN
chain.

The goal4 serve profile is committed non-functional on purpose. The unpacker
and the coverage gate do not exist yet, but the mesh syntax does need
recording: MESH_DEVICE takes a literal tuple, "P300" is not in the plugin's
preset map, and a single visible card would otherwise resolve to (1, N) by
accident.

multica-agent commented

2026-09-05 06:43:07 +02:00

Author

Member

Steps A, B, B2 and C are done — and the plan was missing a step

Progress since the PR opened. All evidence in bench/runs/goal4-nvfp4-unpack-pcc-2026-09-05.jsonl.

A — checkpoint complete

20.56 GB, all 5 shards, verified against the index. 7.5 GB of stale .incomplete blobs from the interrupted first attempt cleaned up.

C — the format is determined, not assumed

property	value
packing	two 4-bit E2M1 per `uint8`, low nibble first
scales	per-group `float8_e4m3`, group_size 16, along the input dim
global scale	scalar `float32`, and it divides

Nibble order is settled by measurement: high-first scores PCC −0.0004 against the FP8 reference where low-first scores +0.9955. Nine projections across MLP, GDN linear_attn and full self_attn, first and last layers: PCC 0.994–0.996, norm ratios 0.984–0.998. That residual is the genuine 4-bit-QAT vs 8-bit-PTQ difference, not unpacker error.

scripts/nvfp4-unpack-check.py gates on two things, and the second is the important one. PCC catches anything that scrambles which value goes where. It cannot catch the global-scale direction — that value is a scalar, so inverting it leaves PCC at ~1.0 with every weight wrong by a constant factor. The norm ratio is the only thing standing between that error and the cards.

B — patch 0009, and the result that matters is not "it loads"

It loads into the same shape: 851 internal keys on both paths, zero unique to either, zero shape mismatches, zero dtype mismatches. Everything downstream of load_state_dict is therefore unchanged — the checkpoint format stops mattering at that boundary. Detection is mutually exclusive both ways.

Laziness holds, which was most at risk: peak RSS 0.23 GB after indexing 2,687 tensors, 2.20 GB after reading five — and that 2.20 is the two 248320×5120 bf16 tensors materialising on demand, not the projections.

B2 — patch 0010, the step the plan did not have

layer.py branches on num_devices > 1, so a (1,1) mesh takes the non-TP modules — and those raise "DeltaNet layer requires the combined qkv_proj weight", because both lazy loaders deliberately keep GDN raw for the TP loaders to shard.

Goals 1–3 could not have found this. It would have surfaced as a crash after a multi-hour cold cache build. as_single_device() supplies the remapped layout without giving up laziness: 851 keys → 947, the conv split reassembles to the original tensor exactly, spec shapes match actual reads.

The generalisation is worth more than the fix: every patch this repo carries (0001–0003, 0007, 0008) targets the TP path. The single-device path is essentially unexercised here and more of it is likely to be wrong. Budgeted for, rather than treated as a surprise each time.

A correction I had to make mid-derivation

I first read LINEAR_Q_DIM/LINEAR_K_DIM = 2048 as hardcoded for the 9B and therefore wrong here. They are not — that is linear_num_key_heads × linear_key_head_dim, 16 × 128 in both models, with V taken as the remainder (6144 here vs the 9B's 4096). The genuinely stale constant there is FULL_ATTENTION_LAYERS = {3…31}, which is dead code.

In flight

Single-card cold build running now on vllm-tt:goal4 — deliberately at today's dtype policy, not QUASAR's coverage yet, so that G4.1 tests the checkpoint and the single-device path with one variable rather than three. Weights are 25.85 GiB against 30.87 usable, so it fits without the bf4 work; step D then buys the context budget.

The log already shows linear_attn.qkv_proj.weight being tilized into a new tensor_cache_bfp8 under P150/ — both patches live on hardware.

## Steps A, B, B2 and C are done — and the plan was missing a step Progress since the PR opened. All evidence in `bench/runs/goal4-nvfp4-unpack-pcc-2026-09-05.jsonl`. ### A — checkpoint complete 20.56 GB, all 5 shards, verified against the index. 7.5 GB of stale `.incomplete` blobs from the interrupted first attempt cleaned up. ### C — the format is determined, not assumed | property | value | |---|---| | packing | two 4-bit **E2M1** per `uint8`, **low nibble first** | | scales | per-group `float8_e4m3`, group_size 16, along the input dim | | global scale | scalar `float32`, and it **divides** | Nibble order is settled by measurement: high-first scores PCC **−0.0004** against the FP8 reference where low-first scores **+0.9955**. Nine projections across MLP, GDN `linear_attn` and full `self_attn`, first and last layers: PCC **0.994–0.996**, norm ratios 0.984–0.998. That residual is the genuine 4-bit-QAT vs 8-bit-PTQ difference, not unpacker error. `scripts/nvfp4-unpack-check.py` gates on **two** things, and the second is the important one. PCC catches anything that scrambles which value goes where. It **cannot** catch the global-scale direction — that value is a scalar, so inverting it leaves PCC at ~1.0 with every weight wrong by a constant factor. The norm ratio is the only thing standing between that error and the cards. ### B — patch 0009, and the result that matters is not "it loads" It loads into the *same shape*: **851 internal keys on both paths, zero unique to either, zero shape mismatches, zero dtype mismatches.** Everything downstream of `load_state_dict` is therefore unchanged — the checkpoint format stops mattering at that boundary. Detection is mutually exclusive both ways. Laziness holds, which was most at risk: peak RSS **0.23 GB** after indexing 2,687 tensors, **2.20 GB** after reading five — and that 2.20 is the two 248320×5120 bf16 tensors materialising on demand, not the projections. ### B2 — patch 0010, the step the plan did not have `layer.py` branches on `num_devices > 1`, so a `(1,1)` mesh takes the **non-TP** modules — and those raise *"DeltaNet layer requires the combined qkv_proj weight"*, because both lazy loaders deliberately keep GDN raw for the TP loaders to shard. **Goals 1–3 could not have found this.** It would have surfaced as a crash after a multi-hour cold cache build. `as_single_device()` supplies the remapped layout without giving up laziness: 851 keys → 947, the conv split reassembles to the original tensor exactly, spec shapes match actual reads. The generalisation is worth more than the fix: **every patch this repo carries (0001–0003, 0007, 0008) targets the TP path.** The single-device path is essentially unexercised here and more of it is likely to be wrong. Budgeted for, rather than treated as a surprise each time. ### A correction I had to make mid-derivation I first read `LINEAR_Q_DIM/LINEAR_K_DIM = 2048` as hardcoded for the 9B and therefore wrong here. They are not — that is `linear_num_key_heads × linear_key_head_dim`, 16 × 128 in **both** models, with V taken as the remainder (6144 here vs the 9B's 4096). The genuinely stale constant there is `FULL_ATTENTION_LAYERS = {3…31}`, which is dead code. ### In flight Single-card cold build running now on `vllm-tt:goal4` — **deliberately at today's dtype policy, not QUASAR's coverage yet**, so that G4.1 tests the checkpoint and the single-device path with one variable rather than three. Weights are 25.85 GiB against 30.87 usable, so it fits without the bf4 work; step D then buys the context budget. The log already shows `linear_attn.qkv_proj.weight` being tilized into a new `tensor_cache_bfp8` under `P150/` — both patches live on hardware.

Grok added 4 commits

2026-09-05 06:43:09 +02:00

goal 4 step C: the NVFP4 unpacker is right, and PCC alone could not have told us 4f0baf8c39

Unpacks one compressed-tensors Linear on CPU and checks it against the same
tensor from the FP8 checkpoint. Nine projections across MLP, GDN linear_attn
and full self_attn, first and last layers: PCC 0.994-0.996, norm ratio
0.984-0.998. That residual is the genuine difference between a 4-bit QAT build
and an 8-bit PTQ build of the same weights.

The format, now determined rather than assumed: two 4-bit E2M1 values per
uint8, LOW NIBBLE FIRST; per-group float8_e4m3 scales at group_size 16 along
the input dim; a scalar float32 weight_global_scale that DIVIDES. Nibble order
is not a guess -- high-nibble-first scores PCC -0.000410 on the same tensor.

Two checks, because one is not enough. PCC catches anything that scrambles
which value goes where. It cannot catch the global-scale direction: that
scalar divides uniformly, so inverting it leaves PCC at ~1.0 with every weight
wrong by a constant factor. The norm ratio is the only thing standing between
that error and the cards, which is why it is a gate and not a printout.

This runs off-device on purpose. A packing error produces a model that loads,
serves, and is subtly wrong -- and the accuracy gate would only find it after
a multi-hour cold cache build.

input_global_scale is deliberately not consumed: those are activation scales
and we quantise activations differently. Dropped explicitly rather than left
to fall through.

goal 4 step B: patch 0009 -- the NVFP4 checkpoint loads, and is indistinguishable downstream 629f91cb5c

Adds is_nvfp4_checkpoint / unpack_nvfp4 / LazyNVFP4StateDict /
load_qwen36_state_dict_nvfp4 to weight_mapping.py, and a second branch in
model_config.load_state_dict. Applies to vllm-tt:src at --fuzz=0.

The result that matters is not "it loads" but that it loads into exactly the
same shape as the FP8 path: 851 internal keys on both, zero keys unique to
either, zero shape mismatches, zero dtype mismatches. Every TP loader
downstream of load_state_dict is therefore unchanged, and the checkpoint format
stops mattering at that boundary. Detection is mutually exclusive in both
directions.

Laziness holds, which was the thing most at risk. Peak RSS is 0.23 GB after
building the index over 2,687 tensors and 2.20 GB after reading five -- and
that 2.20 is the two 248320x5120 bf16 tensors materialising on demand, not the
projections, which were never touched. manifest() returns 851 meta entries. On
a 16 GB host with a 200 GB swapfile this is the difference between loading and
thrashing.

LazyNVFP4StateDict subclasses LazyFP8StateDict and keeps the _spec 5-tuple, so
substate/manifest/shape_dtype/iter/len are inherited rather than duplicated.
Only spec construction and __getitem__ differ.

Placed at the end of weight_mapping.py, after LazyFP8StateDict: the first
version of this patch inserted it before _remap_fp8_key and died at import with
NameError on the base class. The patch applies cleanly either way -- only
running it catches that, which is the argument for the import test being part
of the build and not a manual step.

goal 4: patch 0010 -- one card takes a code path two cards never reach 8b631f4494

layer.py branches on num_devices > 1, so a (1,1) mesh takes the NON-TP modules,
and those call load_gdn_weights, which raises "DeltaNet layer requires the
combined qkv_proj weight". Both lazy loaders deliberately keep GDN linear_attn.*
raw -- fused in_proj_qkv, fused conv1d -- because the TP loaders shard those
themselves. Goals 1-3 ran entirely at TP=2 and could not have found this; it
would have surfaced as a crash after a multi-hour cold cache build.

as_single_device() supplies the remap_qwen36_state_dict layout without giving
up laziness: in_proj_qkv renamed to qkv_proj, conv1d recorded as three slice
specs that slice on read. 851 keys become 947 (+96 = 48 GDN layers x 2 extra
conv keys), the NVFP4 and FP8 single-device key sets are identical, the split
reassembles to the original tensor exactly, and the spec shapes match the
actual reads -- which matters because the warm-cache marker trusts them. Peak
RSS 1.60 GB.

model_config passes single_device=(num_devices == 1), so two cards are
untouched.

I first read LINEAR_Q_DIM/LINEAR_K_DIM = 2048 as hardcoded for the 9B and
therefore wrong here. They are not: that is
linear_num_key_heads x linear_key_head_dim, 16 x 128 in both models, and V is
taken as the remainder -- 6144 here against the 9B's 4096. The stale constant
in that block is FULL_ATTENTION_LAYERS = {3..31}, and it is dead code.

goal 4: record what the first hours found, including the step the plan was missing a023672187

Three findings, none of them in the original work list.

The NVFP4 format is now determined rather than assumed, with the nibble order
settled by measurement (high-first scores PCC -0.0004 where low-first scores
+0.9955). The loader is indistinguishable downstream -- 851 keys, zero
mismatches of any kind against the FP8 path -- which is the result that matters,
because it means nothing below load_state_dict has to know which checkpoint it
got. And a (1,1) mesh takes the non-TP modules, which goals 1-3 never reached;
that cost a patch the plan did not budget for and would otherwise have surfaced
as a crash after a multi-hour cache build.

The generalisation is worth more than the three facts: every patch this repo
carries targets the TP path, so the single-device path is unexercised and more
of it is likely to be wrong. Budget for that rather than treating each break as
a surprise.

Also records a correction I had to make while deriving it: LINEAR_Q_DIM = 2048
looks 9B-specific and is not.

The overlay Dockerfile grows 0009/0010 and an assertion that unpacks two bytes
and checks the E2M1 values, on the same principle as 0008's: a text check
proves the patch landed, only running it proves the behaviour.

Grok added 2 commits

2026-09-05 07:04:04 +02:00

goal 4: 27B runs on ONE card from NVFP4 -- G4.1 and G4.2 met, and goal 3's diagnostic answers differently than either branch it predicted 6a70238594

G4.1: Qwen3.8-27B serves from the QUASAR NVFP4 checkpoint on a single p150a,
(1,1) mesh, one device, coherent output. Cold build 14 minutes, warm restart 8.

G4.2: device weights are 19.764 GiB against 30.87 usable, leaving 11.106 free
-- comfortably inside the 26.0 GiB gate. The goal doc predicted 25.853 and was
6.1 GiB pessimistic, because it derived the budget from the TP=2 cache: the
single-device layout carries no fused prefill mlp.gate_up.swiglu (5.98 GiB
there) and no fused qkvzab/wqkv duplicates. It is genuinely leaner, not merely
differently named, and every KV projection in that section should be re-derived
from 11.106 rather than 8.39. That is now enough for a full 262,144 session
with margin once bf8 KV works.

And this run is the TP=1 diagnostic goal 3 asked for. That doc predicted two
outcomes: unchanged, meaning the GDN chain is the cost, or substantially
faster, meaning the TP=2 all-reduce is. The answer is neither. One card is
2.14x SLOWER (6.94 tok/s, 144.1 ms/token, against 14.864 and 67.3), and MBU is
essentially IDENTICAL -- 30.1% against 32.5%.

So decode scales near-linearly with card count at constant MBU, and the CCL
costs about 7%. What goal 3 measured as ~80% "fixed cost" is fixed with respect
to WEIGHT BYTES, not with respect to hardware: it halves when you add a card.
It is per-card work -- compute or per-op dispatch -- not a serial latency floor
and not DRAM bandwidth. MBU was the wrong instrument for it.

That sharpens step D into a real prediction rather than a hope: if bf4 at
QUASAR's coverage buys much more than its byte share, the cost is matmul
fidelity (bf4 selects LoFi at 16 cycles/tile against bf8's HiFi2 at 32), which
is what QUANTIZATION.md argues and what down_proj alone was too small a share
to show.

One blocker found and worked around. QWEN_SDPA_BF8=1 dies in paged_fill_cache
with input_tensor.dtype() == cache_tensor.dtype(): the non-TP attention module
does not cast the prefill input to the cache dtype. The TP path does, which is
why goal 2 never saw it. KV_BF8 now gates it and defaults off on goal4, at the
cost of 2x KV memory -- a deferred fix, not a closed question.

goal 4 doc: record the measurements, including where sections 3-5 were wrong 102cb8d58d

The budget section was 6.1 GiB pessimistic because it derived from the TP=2
cache; the single-device layout is structurally leaner. The decode expectation
was a miss. And the TP=1 diagnostic answered outside both branches it was
designed around. Keeping the original derivations above and the corrections
below them, rather than editing the predictions to match, because the gap
between them is the part worth reading.

multica-agent commented

2026-09-05 07:04:21 +02:00

Author

Member

It runs. 27B on one Blackhole card, from the QUASAR NVFP4 checkpoint.

G4.1 met. (1,1) mesh, one device, coherent output. Cold build 14 min, warm restart 8 min.

G4.2 met, and the budget was 6.1 GiB pessimistic.

	GiB
§3's prediction (derived from the TP=2 cache)	25.853
measured on one card	19.764
free of 30.87 usable	11.106

The single-device layout is structurally leaner, not merely differently named: no fused prefill mlp.gate_up.swiglu (5.98 GiB at TP=2), no fused qkvzab/wqkv duplication. So every KV number in §4 re-derives from 11.106 rather than 8.39 — at bf8 KV that is ~307k tokens, meaning a full 262,144 session fits with margin instead of being 1.1 GiB short, and the tok_embeddings lever held in reserve isn't needed.

The TP=1 diagnostic answered — outside both branches it was designed around

	tok/s	ms/token	MBU
TP=2	14.864	67.3	32.5%
TP=1	6.94	144.1	30.1%

GOAL-VLLM-SERVING.md predicted "≈14.9 unchanged ⇒ the GDN chain is the cost" or "substantially faster ⇒ the CCL is the cost". Neither: one card is 2.14× slower at essentially identical MBU.

Decode scales near-linearly with card count at constant MBU; the TP=2 CCL costs ~7%. What goal 3 measured as ~80% "fixed cost" is fixed with respect to weight bytes, not with respect to hardware — it halves when you add a card. So it is per-card work (compute or per-op dispatch), not a serial latency floor and not DRAM bandwidth. MBU was the wrong instrument for it.

That converts step D from a hope into a prediction: if bf4 at QUASAR's coverage buys much more than its byte share, the cost is matmul fidelity — bf4 selects LoFi at 16 cycles/tile against bf8's HiFi2 at 32. Which is what QUANTIZATION.md argued and what down_proj alone was too small a share to show.

One blocker, worked around

TT_FATAL @ paged_fill_cache_device_operation.cpp:45:
  input_tensor.dtype() == cache_tensor.dtype()

QWEN_SDPA_BF8=1 makes the cache bfloat8_b, but the non-TP attention module never casts the prefill input to it. The TP path does — which is why goal 2 ran 4×256k on bf8 KV and never saw this. It surfaced only after the full cold build and a successful decode warmup. KV_BF8 now gates it, defaulting off on the goal4 profile at the cost of 2× KV memory. Deferred, not closed: fixing it doubles the context budget.

That is the second instance of the pattern this PR flagged in advance — every patch here targets the TP path, so the single-device path keeps breaking in places two cards never reach. Predicted, budgeted for, and cheaper each time because of it.

Still open

G4.3 — the context ladder. Deliberately capped at 8192 for this run so G4.1 tested one variable, not three.
G4.4 — accuracy vs the bf8 baseline, which is still uncaptured.
Step D — QUASAR's bf4 coverage, now with a specific prediction attached.

Note this run used today's dtype policy, not QUASAR's coverage. The checkpoint is loaded natively; the device dtypes are still ours. Half of "natively" is done.

## It runs. 27B on one Blackhole card, from the QUASAR NVFP4 checkpoint. **G4.1 met.** `(1,1)` mesh, one device, coherent output. Cold build 14 min, warm restart 8 min. **G4.2 met, and the budget was 6.1 GiB pessimistic.** | | GiB | |---|---:| | §3's prediction (derived from the TP=2 cache) | 25.853 | | **measured on one card** | **19.764** | | free of 30.87 usable | **11.106** | The single-device layout is *structurally* leaner, not merely differently named: no fused prefill `mlp.gate_up.swiglu` (5.98 GiB at TP=2), no fused `qkvzab`/`wqkv` duplication. So every KV number in §4 re-derives from 11.106 rather than 8.39 — at bf8 KV that is ~307k tokens, meaning **a full 262,144 session fits with margin instead of being 1.1 GiB short**, and the `tok_embeddings` lever held in reserve isn't needed. ## The TP=1 diagnostic answered — outside both branches it was designed around | | tok/s | ms/token | MBU | |---|---:|---:|---:| | TP=2 | 14.864 | 67.3 | 32.5% | | **TP=1** | **6.94** | **144.1** | **30.1%** | `GOAL-VLLM-SERVING.md` predicted "≈14.9 unchanged ⇒ the GDN chain is the cost" or "substantially faster ⇒ the CCL is the cost". Neither: one card is **2.14× slower at essentially identical MBU**. Decode scales near-linearly with card count at constant MBU; the TP=2 CCL costs ~7%. **What goal 3 measured as ~80% "fixed cost" is fixed with respect to weight bytes, not with respect to hardware — it halves when you add a card.** So it is per-card work (compute or per-op dispatch), not a serial latency floor and not DRAM bandwidth. MBU was the wrong instrument for it. That converts step D from a hope into a prediction: **if bf4 at QUASAR's coverage buys much more than its byte share, the cost is matmul fidelity** — bf4 selects LoFi at 16 cycles/tile against bf8's HiFi2 at 32. Which is what `QUANTIZATION.md` argued and what `down_proj` alone was too small a share to show. ## One blocker, worked around ``` TT_FATAL @ paged_fill_cache_device_operation.cpp:45: input_tensor.dtype() == cache_tensor.dtype() ``` `QWEN_SDPA_BF8=1` makes the cache `bfloat8_b`, but the non-TP attention module never casts the prefill input to it. The TP path does — which is why goal 2 ran 4×256k on bf8 KV and never saw this. It surfaced only after the full cold build and a *successful* decode warmup. `KV_BF8` now gates it, defaulting **off** on the goal4 profile at the cost of 2× KV memory. Deferred, not closed: fixing it doubles the context budget. That is the second instance of the pattern this PR flagged in advance — every patch here targets the TP path, so the single-device path keeps breaking in places two cards never reach. Predicted, budgeted for, and cheaper each time because of it. ## Still open - **G4.3** — the context ladder. Deliberately capped at 8192 for this run so G4.1 tested one variable, not three. - **G4.4** — accuracy vs the bf8 baseline, which is still uncaptured. - **Step D** — QUASAR's bf4 coverage, now with a specific prediction attached. Note this run used **today's dtype policy, not QUASAR's coverage.** The checkpoint is loaded natively; the device dtypes are still ours. Half of "natively" is done.

Grok added 1 commit

2026-09-05 09:12:48 +02:00

measure the FP4 story on our own silicon -- two central claims in QUANTIZATION.md do not survive dd187f50e5

Asked whether there is an upstream PR for native FP4 so we can stop
dequantising to bf16. There is real MXFP4 work upstream -- #46838, #44277,
#42106, #51975 -- and every piece of it targets QUASAR, Tenstorrent's next-gen
architecture, not the QUASAR-QAT model. The names collide unhelpfully. grep
settles it: MxFp4 appears throughout tt_llk_quasar/ and in no file under
tt_llk_blackhole/. Blackhole's format enum has Bfp4 and Bfp4_b and no MX or NV
format at all, so a native NVFP4 matmul is not expressible on this silicon at
any effort, and there is nothing to wait for. Bfp4_b IS the native 4-bit format
here, and the MLP already lands on it -- 8.96 GiB of the measured 19.76 are
already 4-bit on device, not upscaled.

Then measured what fidelity and dtype are actually worth, because
QUANTIZATION.md's "BFP4 at LoFi is exactly 2x BFP8 at HiFi2" is a statement
about this hardware and we own some.

I went in with a hypothesis from tt-metal #39627 -- that LoFi consumes only
about 4 bits of the weight, so a bf8 weight would be truncated and bf4 coverage
would be free. That is refuted. bf8 at LoFi scores PCC 0.999876 against bf4's
0.993162; the bf8 mantissa bits do reach the engine. #39627 says LoFi consumes
the full BFP4 weight, which does not imply anything about BFP8. My prediction an
hour ago that bf4 might buy more than its byte share was wrong, and in the other
direction.

What the measurement does say:

The model already runs LoFi everywhere on the single-device path, so the LoFi
speed is banked and the 2x was never available. At fixed LoFi, bf4 over bf8 is
-28% on one decode shape and 0% on another with identical weight bytes -- the
reduction depth moves it, not the byte count. So bf4 coverage buys footprint,
not single-stream speed, and costs PCC 0.9932 vs 0.9999. Still worth doing on
one card, because footprint is what buys context and concurrency there, but it
has to be argued on those grounds.

And a free win pointing the other way: at decode shapes all three fidelities
take the same time (0.255 / 0.255 / 0.254 ms at M=32), which means the matrix
engine is idle waiting on weight movement and dispatch rather than doing math.
bf8 at HiFi4 is PCC 0.999971 against LoFi's 0.999876, so decode could run HiFi4
for nothing. Prefill could not -- there HiFi4 costs 1.84x.

That idle-engine result is the same conclusion the TP=1 experiment reached from
the other side, and the same mechanism goal 2's batching win exploited.

Grok added 1 commit

2026-09-05 10:24:32 +02:00

goal 4: read the external project's ledger, and probe whether TP can run on one device b463892417

Operator pointed at Thatch-cloud/Tenstorrent.Blackhole-Qwen3.8-27B, which is
weeks ahead on this exact model and pushed today. Two things worth having.

First, independent confirmation. Their demo is 43.1 ms at B=1 and 45.3 ms at
B=8 -- eight times the work for five percent more time. That is the same
dispatch-bound conclusion our TP=1 experiment and our fidelity probe reached
from two other directions, and they quantify it: 68 small ops per layer in GDN,
11.0 ms/step at B=1. They are at 43.1 ms against our 67.3 on two cards, so
about 1.56x ahead.

Second, and worth more, their negative results. Speculative decoding measures
0.65-0.83x on the fused baseline -- slower. That was the obvious next idea for
a dispatch-bound decode and it is now closed without us spending the week.
Their lever J is bf4 read rate, not shipped, which corroborates our own
measurement that bf4 buys 0-28% by shape rather than 2x. Also closed: more
cores, packed gate|up, DRAM-sharded in-projections, CPU offload.

Their shipped levers are recorded with flags and gains. Lever C
(QWEN35_GDN_DECODE_BF16, QWEN35_GDN_STATE_BF16) is env-only and already exists
in our tree -- in gdn/tp.py, so it is inert on the single-device path. That is
the recurring shape of this goal: the optimised path is the TP path, and one
card gives up all of it.

So probed whether the TP path could run on a (1,1) mesh. It cannot as it
stands: all_gather asserts num_devices > 1 and reduce_scatter wants a
neighbour. Not a dead end -- both ops are the identity at TP=1 and the bypass
is mechanical, with most of the 51 CCL references funnelling through
tp_common.py. If it works we inherit lever C, the tuned program configs, our
own 0001-0003/0007, and every future graft, instead of porting each to a path
nobody else optimises.

Probing cost minutes. Finding it the other way costs a 15-minute cold build and
a crash mid-warmup.

Grok added 2 commits

2026-09-05 10:37:04 +02:00

two cards: adopt lever C (GDN bf16 step + state) -- +2.8% for two env vars bb117b8472

Operator's call to refocus on two cards, matching the external project. First
lever off their ledger, and the cheapest: QWEN35_GDN_DECODE_BF16 holds the GDN
decode step at bf16 (high_precision=False on the recurrence op) and
QWEN35_GDN_STATE_BF16 holds the recurrent state at bf16 instead of fp32. Both
already exist in our gdn/tp.py -- they were inert only on the single-device
path, which is exactly why one card was the wrong place to be optimising.

Interleaved A/B in one session, same image and mounts, flag off then on --
their methodology rather than a stock-image control. 14.935 -> 15.359 tok/s,
66.957 -> 65.109 ms/token. Output stays coherent.

The control reproduces the goal-1 baseline of 14.864 to within 0.5%, which is
the check that this measures the flag and not the session.

Neither flag appears in build_variant(), so no cache variant and no cold build.
Their GSM8K for this lever is 57/60, equal to fp32; ours is unverified and
still waits on the accuracy baseline.

two cards: stage lever A (upstream #53587 fused GDN decode) with its rebase mapped a1c1019313

Operator's call: refocus on two cards, where the external project is and where
every optimisation actually lives. Lever C landed already (+2.8%, env only).
Lever A is the big one, +15.5% on its own, and it is a new C++ ttnn op so it
needs the wheel rebuilt.

Rather than guess at a 20-minute build, dry-ran the PR against our fork first.
It does not apply, and the reason is informative: our fork cfx/qwen36-p300-tp2
already carries FOUR of the PR's files -- text_demo.py, gdn/tp.py, tp_common.py
and conv2d.cpp all report "previously applied", which says the fork shares
#53587's lineage. What it does NOT carry is the op itself: no
ttnn.transformer.decode_gated_delta_rule binding, no source directory, no
QWEN_GDN_FUSED_DECODE anywhere in models/. So the rebuild is genuinely
required, and the rebase is two hunks rather than twenty-three files.

Both conflicts are against edits we already understand. ttnn_gated_deltanet.py
@217 collides with our own carried FIR-tap fix -- the (B, k+T, D) one-liner the
Dockerfile asserts and seds -- and that file is where the wiring lives, so it is
the hunk that matters. slice.cpp @325 collides with #53319, which the fork
carries. The 15 new op files apply clean, being new.

Filed under patches/upstream/ rather than patches/tt-metal/, because everything
in the latter is expected to apply at --fuzz=0 and is globbed by a Dockerfile.
That glob already shipped one broken image; a non-applying patch sitting next to
the applying ones is how it happens again.

Recorded one more trap for whoever wires it: Dockerfile.vllm-tt-src exports the
source with git archive HEAD, which does not see an unstaged git apply. Patch in
the wheel stage, commit before the archive, and assert the binding exists in the
final stage -- otherwise the wheel gets the op and /opt/tt-metal-src ships Python
that never calls it.

Grok added 3 commits

2026-09-05 12:33:10 +02:00

lever A: cut #53587 down to the 17 files that matter, and wire the build a7e0bda206

The rebase I mapped last commit turned out not to need resolving. Six of the
PR's 23 files are already in the fork -- it shares the PR's lineage -- and the
two that would have conflicted (ttnn_gated_deltanet.py against our carried
(B, k+T, D) FIR fix, slice.cpp against #53319) are not needed at all. The op
is wired through ttnn_delta_rule_ops.py, and gdn/tp.py:1094 already calls it.
Dropping the six leaves 17 files that git apply --check takes with no
conflicts, verified inside vllm-tt:src.

That reading came from the external project's own graft Dockerfile, which
copies exactly one Python file next to the rebuilt .so. It named the seam.

The build wiring commits the graft before 'git archive HEAD' so the wheel and
the /opt/tt-metal-src overlay cannot disagree, exports the device kernels
(tt-metal JIT-compiles them at run time, so the .so alone is not enough), and
asserts the symbol in the .so and the binding in Python -- neither of which
'it built' would prove.

two cards: measure batched serving for the first time -- 79.5 tok/s aggregate 72d1566784

We have optimised single-stream decode for three goals and never once measured
what the pair does under load. It does 5.53x: 14.358 tok/s at one stream,
79.452 aggregate at eight, for a 31% per-stream cost. That ratio reproduces
the external project's 5.5x on the same shape.

This is the number that answers the operator's standing question about
leveraging the combined 860 GB/s, and it was a config change, not a build.
QWEN_BATCHED_GROUPED=0 is load-bearing above B=4 with short prompts: grouped
batched prefill asserts BH <= ncores, and TP=2 with 48 value heads gives
8 x 24 = 192 against 110 compute cores. goal3 now takes SEQS/CTX and sets the
flag itself rather than leaving it to be rediscovered.

Also recorded: lever E is not available to us. I had found the num_links=1 at
tp_common.py:514 independently and had an A/B running; their plan says that
function has no call sites, our tree agrees, and the run was killed before it
spent twenty minutes measuring an uncalled function. The real link lever needs
a second QSFP-DD cable, which is an operator decision, not a patch.

read the external project's whole programme, not just its README 985a5b67cd

Its optimisation work is on an unmerged branch -- 14,005 lines: a 2,284-line
measured plan, eight lever patches, three custom TTNN ops with kernels, and
the graft Dockerfiles. main is documentation and CI, which is why the earlier
pass only found a results table.

Six of their levers are measured NEGATIVE, and one of those is speculative
decoding at 0.65-0.83x -- the obvious answer to a dispatch-bound decode, and
weeks we now do not spend. Three more are closed at roughly zero, including
matmul core count, which matters because their own patch exposes that
_find_grid caps Blackhole's 110 cores at a Wormhole 8x8. That reads like a
free 3x until you find their number for it: 0.3 ms.

Their v0.77 is not our v0.79, so every claim that depends on dead code or a
hardcoded table was re-checked here rather than inherited. Two changed: lever
E is a second-cable hardware ask for us and not a patch at all, and our fabric
needs no TT_MESH_GRAPH_DESC_PATH at this version.

Grok added 1 commit

2026-09-05 14:03:56 +02:00

lever A measured: +2.3% at B=1, +9.5% at B=8 bb1b59c3f1

Six arms, two interleaved B=1 pairs and a B=8 pair, same image throughout --
the control is the graft with the flag off, never a different image, so the
only difference between arms is the code path taken. Engagement asserted in
both directions: every treatment arm logged the fused op's one-shot line and
every control arm logged none. A control that had silently engaged would have
reported the treatment twice and read as a null result.

B=1  15.539 -> 15.898 tok/s  (+2.31%, -1.45 ms/token)
B=8  89.608 -> 98.100 tok/s aggregate  (+9.48%, -7.73 ms/token)

The external project claims +4.7% and +12.9%. We get about half their B=1
figure and three quarters of their B=8 one, but the SHAPE reproduces exactly
and that is the part that matters: the fusion is worth five times more in
absolute milliseconds at B=8 than at B=1, because the composed small ops stop
being floor-bound once each carries eight rows while the fused op amortises
them. That is the same mechanism their profile describes, arriving at our
numbers independently.

Recorded honestly rather than rounded up: our arm-to-arm spread is 0.5-0.9%
against their 0.1%, so the B=1 result is above noise but not far above it.
The B=8 result is well clear. B=1 TTFT rose with the flag, which is first-call
JIT of the new kernel landing inside the measured request -- prefill does not
take this path -- and is not chased here.

Grok added 3 commits

2026-09-05 14:13:25 +02:00

goal 5: tweak the kernels -- 15.9 -> 18 tok/s single-stream 4e422f0269

The two-card decode deficit is now arithmetic rather than suspicion. Two cards
carry ~430 GB/s each and one decode step reads the whole 27B model once, so the
bandwidth floor for a token is about 36 ms. We are at 63 ms. Over half of every
step is not weight traffic, and the external profile on identical silicon names
what it is: 68 small ops per layer across the 48 GDN layers, 97% device-side.

So the goal tests one thesis to destruction -- that the deficit is op-launch
overhead inside GDN and closes by fusing, not by quantising further, not by a
faster interconnect, and not by speculation. The last three are already
measured negative on this hardware; the doc names them so they do not get
reopened.

The gate is 18 tok/s single-stream, 115 aggregate at 8 streams, GSM8K >= 55/60,
and an A/B with engagement asserted in both directions for every shipped lever.

Honest projection up front: applying our own lever-A haircut (half their B=1,
three quarters of their B=8) to K + 3c + M gives ~17 tok/s, which is short of
the title. Section 4 says where the last tok/s would have to come from and
admits both candidates may come up empty -- including a planned re-measure of
lever H after K, recorded so that rediscovering the 8x8 Wormhole grid
assumption produces a step rather than a second round of excitement.

Order is 3c first as a near-zero-risk rehearsal of the graft-assert-A/B loop,
then K in its four sub-flags rather than as one 1,600-line graft, then M, then
the two profile levers.

lever 3c: stage QWEN_GDN_FUSED_INPLACE as a Python-only overlay 152ce8eb8c

Lever A's graft of upstream #53587 already carries `inplace_state` the whole
way -- nanobind signature, op attributes, C++ writer. Nothing ever passed True.
So 3c is two call sites, 86 lines, and no ttnn rebuild: the overlay builds in
three seconds against vllm-tt:leverA4 where the src Dockerfile is an overnight
job on the fork's compile stage.

That also means 3c needs no second image on cfx-llm2. The flag is env-gated
inside one image, so the control is that image with the flag off -- which is
the control discipline we already use, and it takes the disk pressure out of
this step entirely.

Three build assertions rather than a grep, because 3c can fail in three ways
that all look identical from a benchmark: the wrapper not forwarding the
kwarg, the grafted C++ op predating it (a silent no-op if the base were an
older revision of the PR), and tp.py gating on env alone. That last one is not
cosmetic -- in-place is only legal when init_state IS the persistent buffer, so
an env-only gate would ask for it on a sliced state and corrupt decode at
B < Bmax. The patch guards on identity and the assertion checks for the guard.

The buffer_address() check in the patch is what makes the A/B honest: nanobind
returns a NEW Python wrapper around the same device buffer, so `is` sees two
objects and a lever that silently fell back to the copy path would look like a
clean null result.

Not added to Dockerfile.vllm-tt-src yet -- that is the canonical from-nothing
build and 3c has not been measured. It goes in if the number holds.

Theirs: -0.42 ms at B=1, -3.26 ms at B=8.

bench/ab: check the A/B drivers in, starting with lever A's and 3c's 90003c26f5

Lever A's driver has been living in /tmp on cfx-llm2, which is exactly where a
measurement harness should not live: the numbers in bench/runs are only
reproducible if the thing that produced them is in the repo next to them.

3c's driver differs from A's in one way that matters. A's engagement check is a
single grep for one log line; 3c needs two, because 'requested' and 'happened'
are different facts. The env var can be read and the identity guard can pass
while the op still falls back to the copy path -- and since nanobind returns a
new Python wrapper around the same device buffer, the fallback is invisible to
an `is` check. The patch prints happened=True/False from buffer_address(), and
the driver treats a treatment arm that requested in-place without getting it as
a second control rather than as a result.

Also two passes at B=1 rather than one, and a note in the script saying why the
second may still not settle it: 3c's claimed B=1 gain is 0.42 ms, which is
smaller than the 0.5-0.9% arm-to-arm spread lever A's run actually showed. If
the arms interleave, that is the finding.

Grok added 1 commit

2026-09-05 14:29:19 +02:00

lever K: a warm build tree, and their conv+gates kernel compiles on v0.79 5c2aa4e313

The goal doc named this as K's largest risk: their ops are written against
tt-metal 9f9cd4fd (v0.77.0-rc1) and we are on v0.79.0-dev20260903, with no
upstream PR to smooth the gap the way lever A had. Their own Dockerfile.k warns
the .so is ABI-bound to that rev, and their register_ops.py notes the
transformer ops are unity-built so per-file `namespace cb` blocks collide.

Grafted gdn_conv_gates -- the first of K's four sub-steps -- into our fork's
tree and linked it. It compiled clean on the first attempt, no API drift and no
CB collision, and `ttnn::prim::gdn_conv_gates` is in _ttnncpp.so. One of the
four, so this is not proof the rest follow; but the failure mode the doc
budgeted real rebase work for did not appear where it was most likely to.

One adaptation was needed: register_ops.py anchors its CMakeLists kernel glob
on `gdn_decay/device/kernels/*.cpp`, an op they carry and we do not. Ours
anchors on decode_gated_delta_rule instead. The other three anchors -- both
sources.cmake entries and both nanobind sites -- matched our tree unchanged,
which they only can because we already grafted #53587.

scripts/k-build.sh is the loop that made this affordable. `--target wheel`
tags the src Dockerfile's compile stage, which still carries
/tt-metal/build_Release, and a container off it re-links incrementally: 0.4 s
for a no-op, ~90 s for a fresh op through the transformer unity build, against
~20 minutes for the full image. That matters for more than convenience -- with
a 20-minute cycle the temptation is to graft all four sub-steps at once, and a
combined graft that comes out 3% slow tells you nothing about which quarter was
wrong.

The graft step replaces rather than nests, and syms asserts on nm output,
because an op missing its sources.cmake entry links perfectly and simply is not
there.

Grok added 1 commit

2026-09-05 14:31:16 +02:00

goal 5: close K's rebase risk -- all three ops compile on v0.79 unmodified 339b841f40

gdn_norm_gate and attn_prep link as cleanly as gdn_conv_gates did, so all four
ops (those three plus the grafted decode_gated_delta_rule) are in _ttnncpp.so.
The version gap the goal budgeted real rebase work for is not there.

That moves K's risk entirely into Python: patch_packed.py against the grafted
op, and the wire-k*.sh edits against our tp.py. Those use assert count == 1 on
their anchors, so they fail loudly instead of fuzzing -- which is the failure
mode worth having.

Note this closes M's build risk too, since attn_prep is M's op.

Grok added 1 commit

2026-09-05 14:36:19 +02:00

lever K milestone 1: vllm-tt:k1 builds, conv+gates wired and gated ce96c4bbad

QWEN_GDN_CONV_GATES collapses the GDN conv shift-register, FIR, SiLU and both
gates into one device op -- twelve ops per layer across 48 layers.

Built as an overlay the way the external project builds theirs, not as a full
image rebuild, and the reason that is legitimate is worth stating rather than
assuming: vllm-tt:builder is --target wheel off our own
Dockerfile.vllm-tt-src, and that stage was a CACHE HIT against the build that
produced leverA4. Same fork, same A graft, same compiler, so the .so is
ABI-compatible with the wheel already installed in the base. If the wheel stage
is ever rebuilt cold while the base is not, that breaks -- as a runtime symbol
error, not a build failure, which is precisely what the import assertion is for.

Two traps, both hit on the first attempt and both now documented in the file:

The RUNPATHs differ. The wheel's extension resolves _ttnncpp.so relatively
(/build/lib:...), the build tree's absolutely
(/tt-metal/build_Release/ttnn:...) into a directory this image does not have.
Copied in unmodified it gives ImportError at the first . patchelf
rewrites it to the wheel's; without that the overlay approach does not work at
all.

And the names differ: _ttnn.cpython-310-x86_64-linux-gnu.so in the wheel,
_ttnn.so in the build tree.

The Python wiring is the external project's own wire-k.sh milestone-1 payload
run unmodified against our tp.py -- checked in as patches/k/wire-k1.py with its
output, so the produced file is reproducible rather than a copied artifact.
It applied clean, and the binding name it gates on
(ttnn.transformer.gdn_decode_conv_gates) matches what the nanobind actually
registers. That check matters more than it looks: tp.py gates on hasattr, so a
renamed binding fails CLOSED and the A/B would read as a clean null result
rather than an error.

Not measured yet. This is a build, not a number.

Grok added 2 commits

2026-09-05 14:39:08 +02:00

bench/ab: K1 driver, with three B=1 passes instead of two 3953b01f7c

The 3c run in progress measured a control-to-control spread of 1.2% at B=1 on
this host (15.886 vs 15.697 on identical arms), which is wider than 3c's whole
effect. Two passes cannot separate a sub-1% lever here, so K1 gets three -- and
if three still interleave, that is the finding rather than a direction.

Both arms carry A and C, because K1 replaces part of the path A already fused
and a base without A would credit K1 with A's win. 3c stays OFF in both arms
even though the image supports it: K1 is the variable, and stacking an
unsettled lever underneath would make a null result unattributable.

The engagement check is FATAL on a treatment arm at zero rather than reported
as 0%, because tp.py gates on hasattr() -- a missing or renamed binding fails
CLOSED, producing no error, no speedup, and an arm indistinguishable from a
control.

lever K milestone 2: the packed op patch applies and compiles on our tree too d520b240f8

patch_packed.py rewrites eight files of decode_gated_delta_rule so q/k/v come
in as ONE [1,B,C] tile tensor -- the conv+gates kernel's own output, channels
[q|k|v] head-major -- with the reader gathering head (b,h)'s rows straight out
of that layout. The model's slices, reshapes and repeat_interleaves for GQA go
away.

It applied to our tree unmodified: all eight files, all thirteen of its
asserts, and the result links. That is the second of K's four sub-steps whose
C++ needs no rebase, after milestone 1.

Worth being precise about why this was ever in doubt and why it is not a
surprise in hindsight: our decode_gated_delta_rule is the 17-file cut of
upstream #53587 and theirs is the same PR, so patch_packed is being applied to
the tree it was written against. The version gap between v0.77 and v0.79 lives
in tt-metal around the op, not in the op.

Checked in as the source patch rather than the patched tree, so it stays
reproducible against whatever #53587 cut we carry.

Still no measurement -- milestone 2 needs its Python half (wire-k2.sh) and the
model-side packed entry before it can be run, and milestone 1 is the one with
an image.

Grok added 1 commit

2026-09-05 14:50:08 +02:00

lever 3c measured: +2.7% at B=8; at B=1 the direction holds but the size does not d592817c37

Six arms, engagement proven both ways on every one -- and for 3c that needed
two counts, not one. Every treatment arm logged requested=48 happened_true=48:
all 48 GDN layers asked the fused op to write through, and all 48 got it. Every
control logged 0/0. A treatment that had quietly fallen back to the copy path
would have shown requested=48 happened_true=0 and looked like a null result,
because nanobind returns a new Python wrapper around the same device buffer and
an `is` check cannot see the difference.

  B=8 per-stream  12.329 -> 12.660   +2.68%   -2.12 ms
  B=8 aggregate   98.636 -> 101.279  +2.68%
  B=1 mean        15.791 -> 16.177   +2.44%   -1.51 ms

Adopted. But the two rows do not deserve equal confidence and the record says
so.

B=8 is one pair, and it lands at 0.65x their -3.26 ms -- close to the 0.75x
lever A came in at on the same arm, which is the kind of agreement that makes a
single pair believable.

B=1 is not quantifiable at this sample size, and the honest tell is that it
looks too good. The mean implies 3.59x their -0.42 ms, which is not credible
for a lever whose whole mechanism is one avoided copy. The scatter explains it:
the two controls differ by 1.20% and the two treatments by 2.52%, both larger
than the effect being measured. What survives is the direction -- all four B=1
arms rank treatment above control, and the second pass swung +4.34% -- so the
lever is not harmful and is very likely positive. The number is not evidence.

The driver already anticipated this and said so in a comment before the run;
the K1 driver it seeded takes three B=1 passes instead of two.

Two cards now: 16.18 tok/s single-stream, 101.3 aggregate at 8.

Grok added 1 commit

2026-09-05 15:15:57 +02:00

ledger: audit what of lever K is actually public -- one file is not 5e29a0fdcd

Reading their wire scripts suggests K is copy-and-build. Auditing every path
they reference says otherwise: wire-k2.sh and wire-k3.sh both start by copying
~/kwork/ttnn_delta_rule_ops.py, the packed-capable Python wrapper, and that
file is not in the repository. Everything else -- all four op trees, every
patch script, register_ops.py -- is there.

So K milestones 2 and up cannot be reproduced by copying. Milestone 1 is
unaffected, which is lucky rather than planned: gdn_conv_gates wires straight
into tp.py and needs no wrapper change, so the sub-step we already built and
are measuring right now is the one that did not need the missing file.

The gap is small and the ledger says so rather than just flagging it: the
packed op's nanobind signature is public via patch_packed.py, and the entry
wire-k2 calls maps onto it nearly one to one -- about thirty lines mirroring
the _decode_gated_delta_rule_fused helper #53587 already gives us.

Two other things the audit turned up. patch_state_fast.py is labelled NOT
ADOPTED in its own docstring, a measured negative kept as a record -- not a
lever. And patch_reader_fast.py has no env gate at all: it is kernel-only, so
it needs no Python, but being ungated it cannot be A/B'd inside one image. That
is affordable now only because the data-root move lets cfx-llm2 hold a control
and a treatment image simultaneously -- the first lever whose measurement
depends on that change.

Grok added 1 commit

2026-09-05 15:19:24 +02:00

ledger: map K's real dependency order -- everything past K1 waits on 30 lines 5c3cf6d4f9

patch_reader_fast looked like a free-standing win: no env gate, and it touches
only two kernel files, both JIT-compiled from /opt/tt-metal-src at run time. No
ttnn rebuild at all -- a seconds-long source overlay like 3c.

It does not apply to our tree. It fails on 'gather_scalar zero anchor', which
is text patch_fuse_ng.py introduces, exactly as its own docstring says. Tested
rather than assumed, because the alternative was to schedule it as a cheap
lever and find out during a build.

Charting the rest the same way: only milestone 1 is standalone, and K2 through
K5 plus reader_fast all sit behind the packed path, which sits behind the one
file they did not publish. So the critical path for four fifths of K's claimed
value is about thirty lines of Python wrapper -- not a rebase, not a kernel
rewrite. Worth knowing as a shape rather than discovering one failed patch at a
time.

Nothing here changes K1, which is on the card now.

Grok added 1 commit

2026-09-05 15:31:25 +02:00

ledger: our B=1 arms measure the quantity their plan calls noise 654c9fb5c4

Two A/Bs in a row produced B=1 arms scattering wider than the effect: 3c's
controls 1.2% apart and treatments 2.5%, and K1's second control landing ABOVE
its first treatment. I was treating that as this host being noisy.

It is more specific than that. Their milestone-1 section reports -1.78 ms at
B=1 from exec_sync -- device step time -- and prints the tok/s column beside it
only to dismiss it: 'the tok/s column is host mode's readback noise (4.9-7.1 ms
between arms); exec_sync is the number, as everywhere else in this plan.' Their
tok/s arms read 19.23 / 19.54 / 19.50 / 19.01 and interleave just like ours.

Our harness measures endpoint tok/s. So our B=1 arms are not weak evidence
about these levers, they are evidence about readback jitter, and adding passes
cannot fix a systematic error of that size. The K1 driver's three-pass design
was the wrong remedy for the right observation.

What follows: B=8 is the decisive arm for us -- which is also what their own
sequence says, 'measured at B=8 first' -- and it has behaved cleanly here
(lever A +9.5%, 3c +2.68%) because eight streams amortise the jitter that
dominates a single stream. And a device-time measurement is now the
highest-value improvement to the harness, not the next lever.

Recording their milestone-1 B=8 figure to compare against: -2.40 ms (-4.1%),
independently corroborated by a device microbench at -2.27 ms.

Grok added 1 commit

2026-09-05 15:36:09 +02:00

bench/ab: measure exec_sync, because endpoint tok/s at B=1 is the wrong instrument 89358d69d7

Three A/Bs in and the B=1 arms have never once separated: 3c's controls 1.2%
apart, K1's second control above its first treatment. I read that as a noisy
host and answered it with more passes. It is not a noisy host, it is the wrong
instrument, and their plan says so beside its own numbers -- 'the tok/s column
is host mode's readback noise (4.9-7.1 ms between arms); exec_sync is the
number, as everywhere else in this plan.' Their tok/s arms interleave exactly
like ours.

exec_sync is the host-side time around the traced device step, with update and
readback timed separately so only the model is counted, and their cross-check
puts it within 0.24 ms of DEVICE TRACE FIRMWARE/KERNEL DURATION.

The good news is that it needs no new code. text_demo.py in our own fork
already instruments it, behind QWEN36_DEBUG_DECODE_TIMING=1, and ships the
same two config ids every result in their plan is quoted against: traced_128
for B=1, batched_128_b8 for B=8. This driver is the fifty lines that read it.

It runs the demo as a pytest to completion rather than serving an endpoint --
no readiness poll, no HTTP, no readback path -- which is the whole point.

Engagement discipline is unchanged, plus one addition the endpoint drivers did
not need: a missing exec_sync line is FATAL rather than an empty result, since
a demo that died or an unhonoured flag would otherwise report as a blank arm.

Grok added 1 commit

2026-09-05 15:45:51 +02:00

docs: how the whole thing works, silicon to levers 0110262d7b

The operator asked for the mental model rather than the status: what a Tensix
core is, how a model gets onto two Blackholes, why decode behaves the way it
does, and what we are actually tweaking. Written so future sessions can be
steered on mechanism instead of on lever names.

The organising idea is section 5, and it is the one that explains every
decision in this programme: decode is NOT bandwidth-starved. The bytes say the
floor is ~26-28 ms/token at TP=2; the step is ~55 ms and 97% of it is device
time; and inside that, 11 ms/step is 3,264 GDN op invocations averaging 3.38 us
each regardless of how little data they touch, with 60 of the 68 ops per layer
under 5 us. It is a per-op floor. So fusion is the whole strategy, and
speculative decoding -- the textbook answer to a memory-bound decode -- measured
0.65-0.83x, slower, precisely because the premise does not hold here.

Grounded rather than recalled. The Tensix internals are quoted from tt-metal's
own METALIUM_GUIDE.md in the build container; the JIT claim is checked against
tt_metal/jit_build/ and kernel.cpp's includes; the unity-build collision
against the CMake flag and our op's own namespace cb block; the model dims
against the checkpoint config.json on cfx-llm2; core count, RAM and topology
against the host; GDN's decode path against the real forward_decode in tp.py;
exec_sync against the timing block in text_demo.py. Everything from the
external project is attributed as theirs.

Two corrections to our own docs fell out of writing it. GOAL-KERNEL-LEVERS.md
estimates the bandwidth floor at ~36 ms from ~30 GiB of weights; that
over-counts, because it uses device footprint rather than bytes actually
streamed and the MLP gate/up tensors are bf4. The better-grounded floor is
~26-28 ms -- which strengthens the argument rather than weakening it, since the
gap to reality is larger than we wrote. And the 25.853 GiB weight figure is the
single-card NVFP4 budget and must not be read as a TP=2 per-card number.

The closing section lists what is theirs versus ours and what I could not
verify -- notably that METALIUM_GUIDE's NoC diagram is labelled Wormhole, so
Blackhole's per-core L1 size is not independently confirmed here, and that the
GDN state equation is an intuitive rendering rather than a transcription of the
compute kernel.

Grok added 1 commit

2026-09-05 15:46:52 +02:00

goal 5: correct the bandwidth floor -- I conflated footprint with bytes streamed 162076e1fc

Section 1 derived a ~36 ms floor from 'roughly 30 GiB of weights'. That is the
device footprint, not the bytes streamed per decode step, and it also ignores
that the MLP gate/up tensors are bfloat4_b rather than bf8, so fewer bytes move
than the footprint implies. Counting streamed bytes gives ~26-28 ms, a ceiling
nearer 36-38 tok/s than 28.

Surfaced by the HOW-IT-WORKS.md pass, which did the byte table properly rather
than reusing my round number.

Worth being clear that the correction is not flattering: a lower floor means
the gap between the floor and our measured 63 ms is LARGER, so more of every
step is unexplained by weight traffic. It strengthens the dispatch-bound thesis
this goal exists to test. The projection in section 4 is untouched -- the
bandwidth ceiling was never the binding constraint, which is rather the point.

Grok added 1 commit

2026-09-05 15:47:45 +02:00

lever K1 measured: +3.81% at B=8 -- the closest we have come to reproducing them 5a156afe66

Eight arms. Every treatment logged engaged_lines=48, so all 48 GDN layers took
the fused conv+gates path; every control logged 0.

  B=8 per-stream  12.289 -> 12.757   +3.81%   -2.99 ms
  B=8 aggregate   98.316 -> 102.058  +3.81%

Theirs is -4.1% / -2.40 ms, independently corroborated by a device microbench
at -2.27 ms. We reproduce 93% of their FRACTION. That is much closer than
lever A (which came in at half their B=1 and three quarters of their B=8), and
the reason to trust it is that their two independent methods agree with each
other before ours agrees with either.

Our absolute saving is LARGER than theirs -- 2.99 ms against 2.40 -- while the
fraction is slightly smaller, because our B=8 step is 81.4 ms where theirs is
57.9. Removing 12 ops x 48 layers should be a roughly constant number of
microseconds, not a constant fraction, so a bigger absolute saving on a slower
step is the arithmetic behaving as expected. A tempting reading is that
op-launch costs more on this host and so fusion pays more here; that is a
hypothesis, not a result, and their own profile puts host round-trip at only
1.76 ms of 56.21, so most dispatch cost is device-side and the host CPU is
probably not the explanation.

B=1 is recorded as NOT MEASURABLE rather than as +0.47%. The three passes read
+0.62%, -0.20%, +1.01% against a 2.14% control spread, and we now know why:
endpoint tok/s at B=1 is the readback noise their plan explicitly sets aside.
Their B=1 figure for this lever is -1.78 ms from exec_sync; ours is not
comparable and is not reported as if it were.

Adopted. Two cards: 102.1 tok/s aggregate at 8 streams.

Grok added 2 commits

2026-09-05 16:05:13 +02:00

docs: RECORDS -- the fastest this cluster has actually gone 24bfdabfda

One place for the best measured numbers, so a future session does not have to
reconstruct them from four bench/runs entries and three commit messages.

Admission rule stated at the top: a number enters only if its A/B asserted
engagement in both directions. That keeps projections, borrowed figures and
unengaged arms out by construction.

The single-stream record is marked as the best single ARM rather than a mean,
with the honest figure (~16.0 +/- 0.2) beside it, because our B=1 arms scatter
1.2-2.5% on readback jitter. Publishing 16.38 without that caveat would be
choosing the luckiest arm and calling it a record.

Also records what today's progression was NOT: the 14.36 -> 79.45 jump at 8
streams was not a lever, it was measuring batched serving for the first time on
a host that could always do it. The honest lever total at B=8 is 79.45 ->
102.06, +28.5%.

Prefill is listed as in progress rather than omitted, because its absence is
itself worth seeing: every number in this programme so far is a decode number.

prefill measured for the first time: 3,605 tok/s peak at 16k context e2510869d6

Every number this programme had produced was a decode number. This is the other
half of the serving story, swept 128 -> 32,768 tokens on the full adopted stack
with engagement asserted before any number was read.

  128     0.80 s     160 tok/s
  512     0.54 s     950
  2,048   0.64 s   3,190
  8,192   2.34 s   3,502
  16,384  4.56 s   3,605   <- peak, and the cleanest point
  32,768  9.76 s   3,356

The curve matters more than the peak.

Short prompts are latency, not throughput: at 128 tokens roughly 0.77 s of the
request is engine scheduling, HTTP and one decode step rather than prefill
arithmetic. Reporting 160 tok/s as 'prefill speed' would be reporting our own
fixed overhead. By 16k that overhead has amortised to ~4 ms, which is why that
point is the honest one.

32k comes in 6.9% below the plateau, which is the 16 full-attention layers
starting to cost quadratically -- and is precisely the regime lever D
(QWEN_SDPA_BF8) exists for and where our stack deliberately does not yet enable
it.

One thing this sweep does NOT establish, said plainly in RECORDS: it was taken
with the levers ON, so it is a baseline, not evidence that the decode fusions
are neutral for prefill. The theory says they should be -- prefill is
compute-bound where decode is op-floor-bound -- but a flags-off prefill arm
would be the evidence and has not been run.

Grok added 1 commit

2026-09-05 16:07:20 +02:00

docs: saturation and SRAM -- decode uses 35% of bandwidth and 0.08% of FLOPs a87bc39eeb

Answers the operator's question about chip-level headroom and whether any part
of the model belongs in SRAM, with arithmetic rather than adjectives.

Establishes the hardware constants from source rather than from a diagram.
Blackhole's per-core L1 is 1,572,864 B, read from the Blackhole-specific
dev_mem_map.h and corroborated by the SoC descriptor -- which retires the
'unverified, taken from a Wormhole-labelled diagram' caveat HOW-IT-WORKS.md
carried. 39 KiB of that is system-reserved, per the header's own comment. And
110 cores is now structural rather than merely measured: our cards are
2xharvested, whose compute_with_storage_grid_range is [0,0]-[10,9] = 11x10,
with a further column of 10 Tensix reserved for dispatch.

Total SRAM is 165 MiB per card, which is 1.9% of the 8.69 GiB the model streams
per card per decode step. That ratio is what makes the SRAM question tractable:
weights are never a candidate, so it reduces to the small persistent tensors.

The central finding is a saturation one. Counting streamed bytes at the dtypes
actually used -- bf8 projections, bf4 MLP gate/up/down -- the per-card floor is
21.71 ms against a measured 61.8 ms step at B=1. So 65% of a decode step is
neither weight traffic nor arithmetic, effective bandwidth is 151 GB/s of 430,
and achieved compute is 0.084% of ~491 TFLOP/s with an arithmetic intensity of
2.7 FLOP/byte against a 1,141 FLOP/byte roofline knee. This corrects
GOAL-KERNEL-LEVERS.md's floor a second time, again in the direction that makes
the unexplained gap larger.

On the SRAM question the answer is a qualified yes with an awkward shape. The
GDN recurrent state (B, 24, 128, 128) per card is DRAM-resident today --
reset_state() allocates it with no memory_config -- and is the only tensor that
is small, persistent, and touched every step. But it fits in L1 at B=1, where
it is worth 0.18 ms, and needs 288 MiB at B=8, where it would be worth 1.40 ms.
Partial residency at batch is the realistic form and lands around 0.4-0.6 ms,
which ranks below the packed-path work.

The byte model is validated rather than asserted: it predicts lever 3c's gain
at 1.72 ms against our measured 2.12 ms with no fitting, and it corrects an
assumption I had been carrying -- roughly 80% of 3c's B=8 win was bytes, not op
launches.

Also derives the remaining op-launch budget after A + C + 3c + K1: about 41 of
the original 71 ops per GDN layer, worth ~6.65 ms at B=1 and ~12.79 ms at B=8,
essentially all of it addressed by K2-K5. That is the ranked table's top three
rows and they share one blocker, the unpublished packed wrapper.

Two things I will not overclaim. Lever H's 'launch-bound' framing looks wrong
-- decode matmuls are DRAM-bound, and the real question is whether 33 cores can
issue enough outstanding NoC reads to saturate 430 GB/s, which their pre-fusion
measurement could not have answered. And the 3.38 us per-op average is
demonstrably not uniform, since lever M's measured -0.50 ms is 3x below what it
predicts, so the remaining-dispatch figure is an upper bound.

Unverified items are listed in a closing section, including the runtime L1
allocator base, the attn_output_gate projection shape, and the fact that every
per-op and per-lever constant in the ranked table is theirs rather than ours.

Grok added 1 commit

2026-09-05 16:08:47 +02:00

goal 5: the floor is 21.7 ms on OUR dtypes, not 26-28 on theirs 7eb976b100

Second correction to the same paragraph in one day, subtler than the first. The
26-28 ms figure came from the external project's byte table, which lists MLP
down at bf8. Ours is bf4 -- QWEN36_MLP_DOWN_BF4=1 is in the goal3 profile every
benchmark today ran under, verified in scripts/vllm-tt-serve.sh.

Recounting at the dtypes we actually serve: 8.692 GiB per card per step, floor
21.7 ms, ceiling near 46 tok/s. The two figures are not two analyses
disagreeing; they are two stacks, and the difference is one dtype. Both are
kept, each next to the stack it describes.

The gap widens again. At ~62 ms measured, effective bandwidth is 151 GB/s of
430 (35%) and achieved compute is 0.084% of peak -- roughly two thirds of a
decode step is neither weight traffic nor arithmetic.

Also checks in bench/ab/prefill-sweep.sh, which produced today's prefill curve
and was sitting untracked.

Grok added 2 commits

2026-09-05 16:37:00 +02:00

records: what the hardware is doing while setting them 44a93865b2

A record without a utilisation figure invites the wrong follow-up question. At
16 tok/s the interesting fact is not the number, it is that producing it uses
35% of DRAM bandwidth and 0.084% of the FLOPs -- so the next idea should be
about op count, not about quantisation or clocks.

Carries the caveat that matters for planning: the 3.38 us per-op floor is an
average and is demonstrably non-uniform, since lever M's measured -0.50 ms is
3x below what the average predicts for its 30 ops. The ~6.65 ms of remaining
GDN dispatch is an upper bound, not a forecast.

docs: which layer of the stack is worth optimising, and the order going forward e5cd812504

The operator asked whether further work belongs in tt-metal, TTNN, or higher.
The answer is specific and not the intuitive one, so it is worth a document
rather than a paragraph: almost all remaining headroom is at the TTNN
op-granularity layer, the boundary between the model's Python and the device
operations it issues.

Layer by layer, with the reason each is or is not worth attacking. The
hardware is not the constraint (0.084% FLOP, 35% bandwidth). tt-metal's per-op
floor is not ours to lower, only to amortise. Quantisation is actively
counterproductive for speed because smaller tiles read slower per byte. Serving
is a large multiplier but a different kind of win -- more tokens from the same
steps, not faster steps.

Two things the document is careful about.

The model's Python is not a layer of headroom, it is the authoring surface for
one: host round-trip is 3% of the step, so a faster tp.py is worth nothing
while a tp.py that issues fewer ops is worth everything.

And 3c is written up as the instructive exception rather than smoothed over: it
removes no operations at all, and ~80% of its win was bytes. So the thesis is
'fewer ops AND less data those ops drag', not 'fewer ops'. Getting that wrong
would mis-rank the SRAM work, which is exactly the kind of idea the pure
op-count reading would dismiss.

Ends with the forward order and a four-question test for placing a new proposal
on the map before costing it.

Grok added 1 commit

2026-09-05 16:38:36 +02:00

lever K2: write the packed decode wrapper the external project did not publish aa90b436fa

This is the ~30 lines that everything past K1 was waiting on. wire-k2.sh and
wire-k3.sh both open by copying ~/kwork/ttnn_delta_rule_ops.py, and that file is
not in their repository -- audited this morning, and it is the only input of
theirs that is missing. So it becomes ours to write.

The C++ half stays theirs: patch_packed.py, already verified applying to our
#53587 cut and linking. This is only the Python entry that reaches it, mirroring
_decode_gated_delta_rule_fused which #53587 gives us.

Verified rather than assumed, in three ways.

Every keyword the wrapper passes was checked against the actual nanobind
signature in the built op -- qkv, beta, g, num_k_heads, num_v_heads, head_k,
head_v, scale, initial_state, inplace_state, memory_config. That check is not
pedantry: tp.py gates on hasattr, so a wrong name fails CLOSED, producing no
error, no speedup, and an arm indistinguishable from a control.

The signature was checked against how wire-k2 and wire-k3 actually call it,
including return_row_major, which milestone 3 needs because the norm+gate kernel
consumes ROW_MAJOR directly and the default TILE relayout would be work done
only to be undone.

And it applied, imported and re-ran idempotently inside vllm-tt:k1.

Two behaviours carried deliberately from the unpacked path rather than
reinvented: a one-shot engagement print, so a control arm that silently engaged
is detectable; and a resolve-once-then-warn-and-return-None cache, so an absent
binding falls back instead of raising.

The op is not wired into tp.py yet -- that is wire-k2's surgery, next.

Grok added 1 commit

2026-09-05 18:29:02 +02:00

goal 6: one goal per layer of the stack, sized to its headroom 0a6ccbc649

WHERE-TO-OPTIMISE says where the time goes; this says what we do about it,
layer by layer. Layers 1, 2, 4 get measurements (CCL share, DRAM saturation
curve, op-count probe + post-K profile) because their headroom is either
unknown or not ours to move; layer 3 gets the programme (K2-K5, M, state->L1,
ops/GDN layer 41 -> <=22); layer 5 gets the batch curve; layer 6 gets nothing
for speed and stays a GSM8K gate. Execution order is by expected ms per hour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:38:19 +02:00

lever K2: build the packed q|k|v image; give the exec_sync harness the demo's environment e620a28399

The exec_sync driver ran the pytest demo bare and would have died looking
for Qwen/Qwen3.6-27B online: nothing but vLLM sets HF_MODEL, the mesh and the
caches. It now carries the goal3 profile's environment (bf4 MLP down
included) so device time is measured on the stack every record was set on.

K2 overlays the ttbuild .so pair that has patch_packed applied, ships the
changed reader kernel source, installs our 0012 wrapper, and copies tp.py
wired by the external wire-k2 payload. Staging trap recorded in the
Dockerfile: build_Release/lib/*.so is stale in the warm tree, ninja writes
build_Release/ttnn/*.so -- the lib copy has no packed symbol and would have
shipped as a null-result arm. Assertions check the binding, its signature,
and that the reader on disk is the packed one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:42:17 +02:00

exec_sync harness: override the 300 s pytest timeout and stop matching traced_128k 9b6ffd84d8

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:43:01 +02:00

records: the exact recipe behind every number, and what it does not yet include 88a35ab17a

The 102.06 aggregate was measured with 3c deliberately off; the all-four
stack's decode number does not exist yet. Say so where the number lives.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:44:06 +02:00

lever K2 A/B driver: full stack in both arms, packed flag under test 5f03d65b1a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:44:51 +02:00

goal 6.2: the DRAM saturation probe -- GB/s versus core grid versus dtype 234172ed6d

Decides whether the 21.7 ms floor is reachable from the 33-core grids decode
matmuls run on, and whether lever H was closed for the right reason. Runs
when the device is free; the harness owns it right now.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 18:51:37 +02:00

exec_sync validated: K1 is -1.43 ms at B=1, and vLLM costs ~12 ms/token at one stream bfb8f2a146

The device-time instrument resolves in one pass what three endpoint passes
could not. Its side effect is the larger result: the demo does a token in
~49.8 ms where vLLM does the same token in ~62, so a fifth of single-stream
latency is serving overhead. Recorded as a layer-5 item.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:04:24 +02:00

records: K2 packed q|k|v is +6.3% at B=8 (107.73 aggregate); 3c and K1 do not stack f098799afd

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

multica-agent referenced this pull request

2026-09-05 19:05:57 +02:00

WIP: goals 2 and 3 - long context, and past the external reference #14

Grok added 1 commit

2026-09-05 19:05:59 +02:00

autoresearch image k34m: K3, K4 and M behind their own flags on the K2 base b4c799c3cd

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:06:52 +02:00

autoresearch: queue runner and ledger a3486e7268

The runner measures; the ledger judges. One arm per candidate against the
current best, engagement asserted per flag, adopt at >=1%, park under 0.5%,
discard anything worse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:20:35 +02:00

k34m: move the image assertion into a script (one-line for loops are not Python) c4a1c176ee

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:21:12 +02:00

k34m verify: K2's engagement line lives in the wrapper, not tp.py 28672cc901

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:25:45 +02:00

lever K5: fold norm+gate into the packed recurrence -- wrapper kwargs, wiring, image 6c7b8dcdac

patch_fuse_ng anchors on a cbd namespace that only patch_state_fast (not
adopted) introduces; the copy under patches/k has those anchors adapted to
our cb and applies fully to the K2 tree. 0013 gives our wrapper the four
new kwargs and skips the relayout when the op returns the gated tensor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:26:32 +02:00

ledger: round-1 log, K5 and reader_fast queued, host outage noted b0d2d2e332

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:26:57 +02:00

reader_fast image: two kernels overlaid on k5, no flag, image-vs-image arm e79e6e5b46

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 3 commits

2026-09-05 19:47:19 +02:00

park the Qwen3.8-27B lever programme (operator decision 2026-09-05); effort moves to Ornith MoE 04253b5797

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: Ornith-1.5-35B-A3B MoE on two Blackholes -- up fast, then tweak hard ebf161b55b

Operator pivot 2026-09-05. Carries the 27B programme's instruments and loop,
drops its GDN levers, states inferred targets (60 / 300 tok/s, 8k prefill)
so they can be measured wrong, and lists the MoE lever candidates with
provenance. Phase 0 (scouting upstream presets) is running in parallel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

record the cfx-llm2 hard freeze at the K2 B=1 traced arm; weights for Ornith staging 01f3bfc34c

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:47:58 +02:00

goal 7: the 20-prompt smoke set G7.1 is judged on 07e2f32e60

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 19:50:44 +02:00

goal 7: results file mirroring the external README's four blocks, with a dated history 381b49703a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 6 commits

2026-09-05 20:05:00 +02:00

goal 7 phase 1: Ornith serving image from the Adartras Qwen3.6-A3B bundle 9bd5141bf3

Bundle: Adartras/qwen3.6-a3b-blackhole (HF, 42a7103), single p150, ttnn>=0.77.
Ornith-1.5-35B-A3B is architecturally identical to Qwen3.6-35B-A3B (config diff:
pad_token_id, router_aux_loss_coef, use_cache only). Three deviations, each
its own file: a ttl import shim (tt-lang is cp312-only and pins ttnn 0.74),
an FP8 compressed-tensors loader patch that dequants per-channel and
assembles per-expert tensors into the fused [E,2I,H]/[E,H,I] the bundle
expects, and a hand-written vllm_metadata.json for the plugin's
EXTRA_MODELS_DIR hook.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: phase 0 findings, architecture, roofline and memory plan from the config; phase 1 state dff746312a

Ornith == Qwen3.6-35B-A3B (three trivial config keys differ). Active params
2.94B, of which only a third is experts; 2.75 GB/token at the recipe's
dtypes -> 6.4 ms floor on one card, and the recipe's author sits at 4.5x
that floor -- the 27B's op-floor signature again. Existing-presets table:
the Adartras tt-kernel bundle is the only Qwen3.5-MoE implementation found
anywhere; tt-metal upstream has none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: tt-forge-models Qwen3.6 loader is a verified 2-card sharding blueprint, not a serving path; smoke script

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 4s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 5s

Details

fe1c0782b0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: FP8 loader's gate-first fusion and dequant verified against the bf16 checkpoint (cos 0.9996)

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 4s

Details

746e17f8c8

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: first device command is the bundle's own demo (model-level ms/token), vLLM after

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 4s

Details

169b725ecd

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

goal 7: merge the bring-up branch; floors are 30 TG / 3500 PP (operator), the bundle author's numbers are not the bar b17454a561

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 20:55:09 +02:00

ornith-demo.sh: quote arguments through printf %q so --prompt survives bash -lc 3495e8848c

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 20:55:50 +02:00

goal 7: ladder driver producing the results file's single-stream and B=8 blocks 5c58998663

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 20:56:24 +02:00

goal 7: GSM8K runner matching the external README's protocol (first 60, greedy, 2048 tokens, B=8) 52f7a1acee

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:03:21 +02:00

ornith first light: 40 layers run on one card, coherent output, 33.8 tok/s traced decode 15293f3577

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:16:24 +02:00

goal 7: the bundle's batched MoE path goes dense-256 at B>1 -- lever #1 for aggregate, reproduced at B=8 a5837326aa

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:40:27 +02:00

goal 7: the dual-card knob matrix -- DP/TP/EP tiers, 15 no-code arms, 7 code levers, round-1 queue 7d1d37c7d9

The bundle is single-device (MeshShape(1,1), ReplicateTensorToMesh), so
dual-card today means data-parallel; TP/EP are code. Knobs read from the
bundle's 80+ QWEN36_* env vars with the author's measured notes quoted
where the code carries them, and each arm says what proves it engaged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:41:56 +02:00

ornith scripts: the mapped card is always device 0 inside the container (CARD=1 failed with 'valid IDs are 0..0') ca5ffb18e1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:50:41 +02:00

ornith: vLLM path dies after ~6-9 requests on either card (B1); X280 big-RISC-V scout: idle under tt-metal, do not invest dc14db8ff4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:51:02 +02:00

smoke: ask for the value, not a one-liner -- the model correctly returned an expression and the grader called it a miss 8d879c263f

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 21:57:59 +02:00

ornith: QWEN36_TRACE_REUSE=0 survives the full smoke set 20/20 (B1 suspect confirmed pending control) 3438c30f83

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:00:10 +02:00

ornith B1 attributed: trace reuse across requests hangs the vLLM path (deterministic on card 0, request 6); QWEN36_TRACE_REUSE=0 is the recipe default 22f9777911

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:03:00 +02:00

long-context-bench: count reasoning-stream tokens (Ornith streams thinking under delta.reasoning; the bench saw zero tokens) f6e1b3e19e

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:03:21 +02:00

ladder: warm each rung once, record the second pass (first request per prompt length JIT-compiles) 9f827fe3c5

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:15:10 +02:00

ornith results: first single-stream ladder via vLLM (26.7 tok/s flat to 16k, ~870 tok/s prefill); short-prompt per-length JIT is bug B4 8c62eac131

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:16:21 +02:00

ornith B4 measured: each new short-prompt length costs ~8 s of JIT; repeats 1.2 s 5d3cce8335

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:18:08 +02:00

docs: agent guide to iterating and testing on the TT cards -- machine facts, build loop, measurement discipline, where to focus in the stack 2c973340fa

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:27:58 +02:00

ornith: first full endpoint snapshot — GSM8K 59/60, B=8 dense 50 agg, graceful stop clears B3 4a8a690f55

Correctness gate G7.2 met. Batching at B>1 is the dense-256 path and loses to
DP-2 at B=1, so the next dual-card number is DP-2. Adds the bundle's own
server launcher (the only path with the MTP head wired) and the MTP arm runner.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:29:11 +02:00

bench: round-1 arm runners for Ornith — vLLM no-code knobs and the bundle server's MTP 4aab93d545

Same-card A/B, warm-then-record, graceful stop between arms so no card reset is needed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:39:58 +02:00

bench: arm specs use ',' as the word separator so env names with '_' survive 9ac79fbb2d

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 22:52:13 +02:00

ornith round 1: vLLM dtype knobs are noise-or-worse; the bundle server clears the 30 tok/s floor 7eeff4296e

lm_head bf8, KV bf16 and expert-down bf8 all lose or tie against the recipe defaults, matching
the roofline. The bundle's own server decodes at 33.1 tok/s where vLLM gets 27.9 on the same
card and recipe, so the serving layer is the single-stream lever — but it stalls on ~4k prompts
(new bug B5, bisect script added).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:03:05 +02:00

ornith: B5 is B1 in the bundle server — trace reuse across requests wedges the card on a long prefill d6907f1031

Bisected on card 0: with the server's default trace reuse the first prompt over the 2048 long-prefill
threshold dies in the first chunk with an MMIO per-op timeout; with QWEN36_TRACE_REUSE=0 the same
prompts prefill in 17-21 s. The server launcher now defaults to TRACE_REUSE=0, and reset-card.sh
scripts the reset-settle-probe loop a wedged card needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:13:58 +02:00

results: bundle server meets the 30 tok/s single-stream floor on one card; MTP blocked on tt-lang 49a3c3e11f

33.2 / 33.3 / 32.2 tok/s at 128 / 1k / 4k with TRACE_REUSE=0. The MTP verify step is a chained
tt-lang launch, so speculative decode waits on lever 2.2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:15:37 +02:00

results: first dual-card number — DP-2 of the bundle server, 66.4 tok/s aggregate, no per-card loss f271504511

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:21:30 +02:00

ledger: prefill knobs round 1 — chunk 2048 trims 4k TTFT 7 %, traced prefill flat; arm runner keeps server logs b12c9289d0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:27:43 +02:00

prefill: the whole 4k TTFT is dense-256 MoE DRAM traffic; grouped top-k prefill is the G7.4 lever 37736998ff

Chunk 2048 adopted (-7 %), chunk 1024 crashes chunked SDPA (B6), traced prefill flat. No env knob can
reach 3,500 tok/s because prefill moves 256 experts' worth of activations per token; lever 2.8 added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:29:01 +02:00

ledger: bundle server passes the same 59/60 GSM8K as vLLM; stable-prefill toggle is noise 665f76b113

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:31:56 +02:00

ledger: un-adopt prefill chunk 2048 (crashes at 16k, B6 generalised); sparse prefill +69 % slower as documented 33c85908f9

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:34:29 +02:00

results: bundle server ladder reaches 16k at 31.6 tok/s, 19.9 s TTFT e5d207dfaf

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:47:00 +02:00

prefill: measured breakdown replaces the MoE-DRAM estimate — 55 % of TTFT is host dispatch; traced prefill never engaged upstream a158d1cbe5

Phase profile at 4k: GDN 1.0 s, MoE 0.8 s, attention 0.4 s of a 5.3 s TTFT. The server's traced
prefill silently fell back to eager because no buckets were ever captured; a patch overlay
(ornith/patches/server-prefill-buckets.py) adds the setup call and is being measured. All seven
off-by-default decode knobs are within ±0.6 %.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-05 23:57:44 +02:00

results: DP-2 at 16k — 63.7 tok/s aggregate; traced prefill needs a >1.2 GB trace region for buckets to 8k dccd83d582

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-06 00:20:01 +02:00

traced prefill: engages after the patch but a padded 1k bucket wedges the device (B7) — parked 3badc276ed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-06 00:23:26 +02:00

goal 7 state, results history and agent guide: day one closes with decode floor met, prefill floor open, all flags spent 1c8972cef0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-06 16:20:38 +02:00

prefill programme: the budget shows flags cannot reach 3,500 tok/s; staged plan is chunk-level trace, GDN device loop, MoE bytes 646af120ce

Measured phase profile plus the bundle's own dense-MoE cost model give a hard
ceiling of ~1,780 tok/s for dispatch removal alone, so the plan sequences
host-dispatch work, a GDN device-loop kernel adapted from upstream, and MoE
weight-traffic cuts, each behind simulator-first and token-identity gates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-06 18:36:58 +02:00

Ornith prefill: record 28-arm lever sweep (O22-O28) and grouped-MoE finding f9170c9a56

Grouped MoE prefill measured at 462/493 tok/s @4k/8k vs 763/827 default:
a -40% regression, not a win. Sparse decode alone is neutral. No other
prefill flag moves the number. Data + script added to bench/runs and
bench/, sweep bullet added to the programme doc, ledger rows O22-O28.
Unexplained null arms (gdn_fused, gdn_dtype_fp8, gdn_verify_chain,
attn_dtype_fp8) flagged for the run-2 card-wedge investigation.

Grok added 1 commit

2026-09-07 09:18:31 +02:00

27B production programme: definition of done, operator decisions, unit and gateway shape, soak gate 3d537ab65c

The record recipe exists but nothing owns it: no unit, no digest pin, no
route, and the box is at 0 B free on the model store. This sequences the
work from host health through a tenstorrent-serving module, a static Bifrost
backend, a wedge watchdog and a 24 h soak, with the operator decisions that
must come first (bring-up-only ruling, card ownership vs Ornith, disk, model id).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok added 1 commit

2026-09-07 17:13:57 +02:00

goal 8: DSpark D=7 speculative decode for the 27B — three legs (verify cost, on-device drafter, acceptance on our quant) with the arithmetic that shows 50 tok/s needs all three fcecdd7b7e

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Grok referenced this pull request

2026-09-25 10:06:18 +02:00

1-card target: ThinkingCap-27B on ONE P150a (TP=1) - context ladder, accuracy, bf4 coverage, batching #69

Grok commented

2026-09-25 10:06:28 +02:00

Owner

Closing: the content of this PR is merged on master, and its base agent/claude-auth/goal-2-3 is dead (archived as tag archive/agent/claude-auth/goal-2-3). G4.1 and G4.2 were met: 19.76 GiB of weights on one card and 144 ms/token at TP=1 (bench/runs/goal4-single-card-nvfp4-2026-09-05.jsonl).

The open 1-card goals move to #69: G4.3 context ladder, G4.4 accuracy, and Step D QUASAR bf4 coverage. The branch agent/claude-auth/goal-4-nvfp4-single-card is left in place because it carries one unmerged doc.

Closing: the content of this PR is merged on master, and its base `agent/claude-auth/goal-2-3` is dead (archived as tag `archive/agent/claude-auth/goal-2-3`). G4.1 and G4.2 were met: 19.76 GiB of weights on one card and 144 ms/token at TP=1 (`bench/runs/goal4-single-card-nvfp4-2026-09-05.jsonl`). The open 1-card goals move to #69: G4.3 context ladder, G4.4 accuracy, and Step D QUASAR bf4 coverage. The branch `agent/claude-auth/goal-4-nvfp4-single-card` is left in place because it carries one unmerged doc.

Grok closed this pull request

2026-09-25 10:06:28 +02:00

Grok commented

2026-09-25 10:06:31 +02:00

Owner

agent-gateway closeout (closed): cleared agent:* labels.
Hermes kanban card (if any) is left for Hermes to archive/complete.

agent-gateway closeout (closed): cleared `agent:*` labels. Hermes kanban card (if any) is left for Hermes to archive/complete.

Grok referenced this pull request from a commit

2026-09-25 10:08:01 +02:00

docs: point the 1-card items at the new issue #69

Grok referenced this pull request

2026-09-25 10:09:04 +02:00