bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

WIP: goal 7: Ornith-1.5-35B-A3B MoE bring-up (Adartras bundle, FP8 loader, ttl shim) #16

Closed
multica-agent wants to merge 0 commits from agent/claude-auth/ornith-moe into master
pull from: agent/claude-auth/ornith-moe
merge into: bitpartner:master
Conversation 2 Commits - Files changed -
multica-agent commented 2026-09-05 19:57:04 +02:00
Member
Copy link

Tier T1 (serving image + docs; no host config).

Goal 7 bring-up: docs/GOAL-ORNITH-MOE.md, the Ornith-1.5-35B-A3B MoE (27B programme dropped for the MoE).

Phase 0 scouting + Phase 1 image. Recipe: HF Adartras/qwen3.6-a3b-blackhole tt-kernel vLLM bundle (single p150, 32.5 tok/s external), loaded via our plugin's EXTRA_MODELS_DIR hook. Deviations: ttl (tt-lang) import shim (cp312-only wheel, pins ttnn 0.74, so GDN decode runs the bundle's ttnn scan fallback), FP8 compressed-tensors loader patch with per-expert -> fused assembly (verified offline: bf16 fused vs FP8 dequant cosine 0.9996/0.9997), hand-written vllm_metadata.json, serve/smoke scripts. Image vllm-tt:ornith built and import-verified on nixos.

Device run is done (2026-09-06 00:40), not pending. All measured on cfx-llm2, 2× Blackhole p150a, (1,2) mesh:

criterion state
G7.1 up + smoke met — 20/20 both cards, both serving paths
G7.2 GSM8K ≥55/60 met — 59/60 (identical miss) on vLLM and bundle server
G7.3 decode ≥30 tok/s single stream met — 33.2 / 33.3 / 32.2 / 31.6 at 128/1k/4k/16k (bundle server); stretch 60 open
G7.4 prefill ≥3,500 tok/s open — ~770-820 tok/s per card at 4k-16k
G7.5 ledger + results with history met — RESULTS-ORNITH.md, LEDGER.md O1-O21

DP-2 both cards: 66.4/66.1/64.5/63.7 tok/s aggregate at 128/1k/4k/16k, zero per-card loss. All 21 no-code knobs spent (none adopted beyond the serving-path switch); everything from here is code.

Open levers, in order: traced prefill (B7 wedges on a padded 1k bucket), fused GDN decode kernel (needs tt-lang or a hand kernel; also unblocks MTP), grouped top-k MoE prefill (lever 2.8), sparse batched MoE (lever 2.1).

Branch was cut from agent/claude-auth/goal-4-nvfp4-single-card and carries its parked 27B commits; the Ornith diff is ornith/, docker/ornith, docs/GOAL-ORNITH-MOE.md.

Tier T1 (serving image + docs; no host config). Goal 7 bring-up: docs/GOAL-ORNITH-MOE.md, the Ornith-1.5-35B-A3B MoE (27B programme dropped for the MoE). Phase 0 scouting + Phase 1 image. Recipe: HF Adartras/qwen3.6-a3b-blackhole tt-kernel vLLM bundle (single p150, 32.5 tok/s external), loaded via our plugin's EXTRA_MODELS_DIR hook. Deviations: ttl (tt-lang) import shim (cp312-only wheel, pins ttnn 0.74, so GDN decode runs the bundle's ttnn scan fallback), FP8 compressed-tensors loader patch with per-expert -> fused assembly (verified offline: bf16 fused vs FP8 dequant cosine 0.9996/0.9997), hand-written vllm_metadata.json, serve/smoke scripts. Image vllm-tt:ornith built and import-verified on nixos. **Device run is done (2026-09-06 00:40), not pending.** All measured on cfx-llm2, 2× Blackhole p150a, (1,2) mesh: | criterion | state | | --- | --- | | G7.1 up + smoke | met — 20/20 both cards, both serving paths | | G7.2 GSM8K ≥55/60 | met — 59/60 (identical miss) on vLLM and bundle server | | G7.3 decode ≥30 tok/s single stream | met — 33.2 / 33.3 / 32.2 / 31.6 at 128/1k/4k/16k (bundle server); stretch 60 open | | G7.4 prefill ≥3,500 tok/s | open — ~770-820 tok/s per card at 4k-16k | | G7.5 ledger + results with history | met — RESULTS-ORNITH.md, LEDGER.md O1-O21 | DP-2 both cards: 66.4/66.1/64.5/63.7 tok/s aggregate at 128/1k/4k/16k, zero per-card loss. All 21 no-code knobs spent (none adopted beyond the serving-path switch); everything from here is code. Open levers, in order: traced prefill (B7 wedges on a padded 1k bucket), fused GDN decode kernel (needs tt-lang or a hand kernel; also unblocks MTP), grouped top-k MoE prefill (lever 2.8), sparse batched MoE (lever 2.1). Branch was cut from agent/claude-auth/goal-4-nvfp4-single-card and carries its parked 27B commits; the Ornith diff is ornith/, docker/*ornith*, docs/GOAL-ORNITH-MOE.md.
The endpoint served 256 streamed tokens at 14.368 tok/s single-stream decode
on Qwen3.8-27B-FP8 across both p150a at TP=2. Goal 1 asks for >= 10, so this
passes; decode-bench.py printed FAIL only because it still hardcoded the 18
that the operator retired earlier the same day. The threshold is now --target
with a default of 10, so the script measures and the goal document decides.

Recorded with the distribution, not just the mean, because the shape is the
interesting part: p50 69.8 ms, p99 82.0 ms, max 88.4 ms. A p99/p50 of 1.18
means no periodic stall -- the ~51% of DRAM bandwidth we are leaving unused is
systematic per-token overhead, which is what goals 3's levers have to attack.
Goal 1's 14.368 tok/s is ~49% of the measured DRAM bandwidth, and decode is
already traced -- so the remaining overhead is not dispatch and the cheap MBU
levers are spent. The way past the external reference's 16.0 tok/s is to make
the divisor smaller, not the utilisation higher.

down_proj is the obvious first cut: 5.704 G params, the last tensor of MLP
size still at bfloat8_b now that gate/up are bfloat4_b. At bfloat4_b it drops
2.65 GiB from the 27.49 GiB per-token read, moving the 100%-MBU ceiling from
29.1 to 32.2 tok/s and freeing 2.65 GiB of device DRAM that goal 2's 4x256k KV
budget wants. Two published NVFP4 recipes for this model quantise down_proj,
so this is the evidence-led candidate rather than the first one to hand.

Patch 0008 gates it on QWEN36_MLP_DOWN_BF4 -- upstream's 'down: bfloat8_b
(accuracy)' is a deliberate choice and it has to survive a greedy comparison
before it becomes our default. The second hunk is the one that would have
wasted the whole experiment: the MLP cache file name carries no dtype, so
without adding the gate to build_variant() the warm cache would have handed
back the old bfloat8_b tensors while reporting a complete build, and the
benchmark would have shown no change for the wrong reason.

The overlay Dockerfile exists because the wheel stage is an overnight compile
here and our build cache was pruned; it applies Python patches onto an existing
image, and refuses to produce one where the patch did not land.

scripts/vllm-tt-serve.sh finally puts the working invocation in the repo with
the three profiles, which is goal 1's third condition.
The first overlay build succeeded and produced a broken image. It globbed
patches/tt-metal/*.patch, but 0001-0004 are already merged into the fork
branch the base image is built from, so it re-applied them onto a tree that
already had them. They matched cleanly in neither direction, patch(1) took
them with fuzz regardless, and the result lost the `Mode` import that 0003
touches. The container then died at import with `NameError: name 'Mode' is
not defined` -- nowhere near the dtype change under test, and only after a
device open and several minutes of engine init.

Three changes, each aimed at that failure rather than at its symptom:

- the COPY names the one patch that is not in the base, so 'which patches does
  this image add' is answered by reading the Dockerfile;
- --fuzz=0 --forward, so a patch that no longer matches its context fails the
  build instead of being guessed at;
- a real import of qwen36_vllm plus an assertion that the gate reaches
  build_variant(). The grep proves text landed; only an import proves the tree
  still parses, and only the assertion proves the cache will be invalidated.

0008 is regenerated with diff -u against the pristine tree inside vllm-tt:src
instead of being hand-written, so it now applies at zero fuzz. Verified in the
image: import clean, gate True.

The transferable part: `patch -R --dry-run` is not a reliable already-applied
test on a tree the patch was rebased across.
Both existed only as sentences inside goal 1's 'explicitly not in goal 1'
list, which was fine while goal 1 was unmet and is not fine now that it is.
Each now has a gate that passes or does not, the arithmetic that says whether
it is reachable, and an ordered list of what is likely to stop us.

Two findings from writing them down are worth more than the prose:

Goal 2 is bounded by bf8 KV, not by cleverness. 4 x 256k is 1.05 M tokens of
KV; at bfloat16 that is 64 GiB against 64 GiB of total device DRAM, and at
bfloat8_b it is 32.5 GiB. QWEN_SDPA_BF8 is therefore load-bearing rather than
an optimisation -- and the ~1.4 GiB of margin it leaves is thin enough that
goal 3's down_proj change, which returns 2.65 GiB of DRAM, belongs first. That
is the ordering argument between the two goals, and it runs the opposite way
to their numbering.

Goal 3 cannot be one change. Decode is ALREADY traced -- qwen36_vllm.py
captures at pos 0 and replays per bucket -- so the dispatch lever a naive
implementation would start with is spent, and goal 1's p99/p50 of 1.18 says
there is no stall left to remove. What remains is the divisor, and down_proj
at bfloat4_b lands at ~15.9 tok/s against a > 16.0 gate. Stating that up front
beats discovering it after the run.

Also records goal 2 as a pooled 1,048,576-token budget rather than a 4x256k
product, because QWEN36_MAX_TOKENS_ALL_USERS is how the model itself expresses
concurrency x context, and the pool is simultaneously 4x256k, 8x128k and
16x64k.
Three corrections, all found by running the thing rather than reading it.

The gate was inert where it mattered. 0008 changed only load()'s single-device
path; on a two-card mesh load_mlp_weights returns early through tpc.shard_w,
which has its own two down_proj call sites. The image built, imported,
asserted its own gate, loaded 64 layers and served -- while every log line
read mlp.down_proj.weight.tp_dtype_BFLOAT8_B. _down_dtype is now hoisted to
the top of the function and the overlay counts four occurrences, so one
patched site out of three fails the build instead of producing a null result
that looks like a finding.

The tensor cache does not collide across dtypes, contrary to what the first
version of this patch and the goal 3 section both claimed. ttnn's as_tensor
appends the dtype to the filename it writes, so bf8 and bf4 are separate
files and the flag is warm in both directions once each set exists. The
build_variant hunk stays -- the marker certifies a variant's file set is
complete, and bf4 is a different set -- but the reasoning behind it was wrong
and is corrected in place rather than quietly kept.

decode-bench.py referred to args.target where the namespace is called a, so
the threshold change I made two commits ago crashed the script on first use.
Fixed, and the fix is why there is a second baseline number at all.

That second number is the useful one: 14.418 tok/s against 14.368 earlier,
0.3% apart, after a full container teardown and a warm start from the tensor
cache. Goal 1's third condition -- reproducible, comes back without
hand-holding -- is now measured rather than assumed.
long-context-bench.py answers the two questions decode-bench.py cannot: does
the KV allocation hold at depth, and what does depth cost. You cannot infer
either from 4k on this architecture -- 16 of 64 layers scale with context and
48 do not -- so it builds a real prompt of a real length and runs N of them
concurrently. TTFT with one stream and no queue IS the prefill measurement,
which is the number goal 2 is most likely to be judged on, since prefill at
256k on four Zen 1 cores is expected to be minutes.

It ladders on purpose. Each distinct prefill width compiles on first use, so
32k -> 128k -> 256k costs more wall clock than 256k alone, and buys a failure
that is cheap to read instead of one that costs an hour to reproduce.

The doc correction matters more than the script. Goal 3's section claimed the
MLP tensor cache would silently hand back bfloat8_b tensors under a bfloat4_b
flag because the cache file name carries no dtype. That was wrong -- ttnn's
as_tensor appends dtype and layout to the name it writes, so the two dtypes are
separate files that coexist, and switching the flag is a warm start in both
directions. The design is better than I assumed, and the dtype sweep is
correspondingly cheap: each candidate materialises once, ever.

Replaced with the trap that was real: at TP=2 load_mlp_weights returns early
through tp_common.shard_w, so the branch that reads naturally when you skim
the function is not the branch that runs. That one produced a serving model
whose logs said BFLOAT8_B under a flag asserted at build time.
The bf4 down_proj run loaded all 64 layers and then refused to start:

  RuntimeError: Warm build generated 64 new tensorbins from placeholders
  (e.g. layers.0/mlp.down_proj.weight.tp_dtype_BFLOAT4_B_layout_TILE...);
  they were deleted. The marker did not cover this build.

That is patch 0005's guard doing exactly what it was written for, and it is
worth being clear that it saved the experiment rather than blocking it. The
warm path substitutes dataless NaN placeholders for device weights, because on
a warm start the real values are already tilized on disk. A dtype change
creates filenames that do not exist yet, so the warm path would have happily
tilized the placeholders and served a model of NaNs -- which at best crashes
and at worst produces plausible-looking garbage.

COLD=1 sets TT_TRANSFORMERS_FORCE_MODEL_LOAD=1, needed exactly once per new
tensor-cache variant. After that the bf4 set is on disk and, because ttnn keys
cache files on dtype, both dtypes stay warm and switching between them is
free. The comment in the script says all of this, since the error message is
the kind that reads like a defect at 2am.
The two kinds were undifferentiated, which made every patch look like it
needed the overnight wheel compile. 0005-0008 are Python and land in the
image's last layer in seconds; only 0004 and the two upstream PRs touch the
wheel. Recording which is which is what makes the overlay Dockerfile a
legitimate tool rather than a shortcut, and it comes with the constraint that
keeps it honest: both Dockerfiles must list the same Python patches.

Also records the three failure modes from today, because each cost real time
and none is guessable: globbing the patch directory onto a fork-built tree
re-applies the patches already merged into the fork branch; hand-written hunk
headers apply with fuzz and then fail under --fuzz=0; and a dtype change needs
exactly one COLD=1 build or the warm path tilizes NaN placeholders into the
new filenames.
Every MBU figure in this repo took 27.49 GiB from the external report. Summing
our own tensor-cache files by name group says that is wrong, because two of
the largest cached tensors are not read per token:

  mlp.gate_up.weight.swiglu.tp   5.98 GiB  packed [gate|up] for the fused
                                           PREFILL agmm -- the source says
                                           decode keeps w1/w3
  tok_embeddings.weight          2.37 GiB  a row lookup, not a matmul

The corrected read is 20.17 GiB, and it moves both numbers in opposite
directions: the 100%-MBU ceiling rises from 29.1 to 39.7 tok/s, and our
measured 14.42 tok/s is 36% of DRAM rather than 49%.

That is better news and worse news, and the worse half is the interesting one.
More headroom exists than documented. But a decode that is ALREADY TRACED,
with a p99/p50 of 1.18 and no episodic tail, sitting at 36% MBU, has something
systematic in front of every token -- which is not the conclusion the previous
revision drew when it said the dispatch levers were spent. Three candidates
are now written down with the evidence for each: host round-trips (_ondev_argmax
is False, so logits cross PCIe Gen3 x4 every token), M=1 matmul efficiency on a
path tuned for 1 and 4 cards, and the per-layer CCL at a TP=2 that falls back
to TP=4 constants.

It also changes the goal 3 projection from a miss to a pass: at a constant 36%
MBU, down_proj at bfloat4_b lands at 16.6 tok/s against the > 16.0 gate rather
than the ~15.9 the old divisor predicted. Whether MBU holds constant is the
open question, which is the argument for measuring instead of projecting.
14.864 tok/s against a 14.418 baseline. The read shrank 13.2% and the time
shrank 3.0%, so MBU FELL, from 36.0% to 32.5%. That is the opposite of what a
bandwidth-bound decode does when you hand it fewer bytes, and it is the most
useful number produced today.

Fit t = fixed + read/B across the two points: ~54 ms of the 67 ms per token is
NOT streaming weights. Roughly 80% fixed cost. The consequence is arithmetic
rather than opinion -- moving every remaining bfloat8_b group to bfloat4_b
(qkvzab 4.02, GDN out 1.49, lm_head 1.26, wqkv 1.16, wo 0.50 GiB) buys 3.97
GiB, about 3.1 ms on the measured slope, about 15.6 tok/s. Still short of 16.0.
So no reachable dtype policy closes goal 3's gap, and continuing down that
path would have burned a cold build per candidate to find that out one
candidate at a time.

Where the fixed cost is NOT: I had written host logits readback as the leading
suspect on the grounds that model.py sets _ondev_argmax = False. That is the
wrong read of the code -- the vLLM decode path calls _forward_decode with
sharded_lm_head=True, which takes the same pre-gather branch, so the
all-gather and full-logits readback are already skipped. Removing a suspect
counts.

Also adds dtype-accuracy-check.py, which scores single next-token predictions
across 24 strongly-determined prompts. The obvious design -- echo the prompt
and diff per-token logprobs -- is unavailable: the TT plugin answers
'Not yet supporting prompt_logprobs on tt'. And diffing two greedy generations
is worse than useless: they diverge at the first perturbed token and every
token after differs by construction, which scored this change at 6% agreement
when nothing had been shown to be wrong with it.
Replaces the projection with the measurement. down_proj at bfloat4_b cut the
per-token read 13.2% and bought 3.1%, and MBU fell from 36.0% to 32.5% -- a
bandwidth-bound decode handed 13% fewer bytes does not behave that way.
Fitting the two points puts ~54 ms of the 67 ms per token outside the weight
stream, and halving EVERY remaining bfloat8_b group buys ~3.1 ms on that
slope, landing at ~15.6 tok/s against a 16.0 gate -- while optimistically
assuming lm_head and the attention projections tolerate 4 bits, which they
probably do not.

Better to write that down now than to spend one cold build per candidate
discovering it one candidate at a time.

Records where the cost is not, since a ruled-out suspect is worth keeping:
_ondev_argmax reads False, but the vLLM path calls _forward_decode with
sharded_lm_head=True and takes the same pre-gather branch, so logits readback
was never happening. Decode is traced with no episodic tail, so it is not
per-op dispatch either.

The hypothesis that survives is architectural: 48 of 64 layers are Gated
DeltaNet, their per-token work is a fixed-size recurrent state update, and
fixed-size means latency-bound -- small tensors in a serial chain, nothing to
amortise. ~1.1 ms per GDN layer is the right order. If that holds, the
property that makes 256k context affordable is the same one that makes
short-context decode slow, which is a genuinely interesting trade and not one
either goal anticipated. Flagged as needing a profile before more code.

down_proj bf4 stays on for goal 2 regardless, on memory grounds: 2.65 GiB of
DRAM is the difference between ~0.3 and ~3 GiB of KV margin at 4x256k.
32k x 1: prefill 9.18 s, decode 14.693 tok/s, 513 KV blocks allocated.

Both numbers correct something. The goal 2 section listed prefill time as the
most likely blocker, on the strength of goal 1's 6m06s prefill warmup at chunk
2048. That was a bad generalisation: the warmup is compilation, and steady
prefill runs at ~3270 tok/s, which puts 256k at roughly 80 s. The risk list had
its first item wrong.

And decode at 32k is 1.1% below decode at 4k -- 14.693 against 14.864 -- for
eight times the context. That is the Gated DeltaNet structure behaving exactly
as VRAM-BUDGET.md said it would, with only 16 of 64 layers carrying a
context-scaling cache. It also sits oddly next to goal 3's finding: the same
architecture that makes context nearly free is the one leaving decode at 32%
MBU.
Prefill time was listed first on the strength of goal 1's 6m06s warmup at
chunk 2048. The 32k rung measured 9.18 s for the same work at eight times the
length, because the warmup is compilation and not throughput -- so prefill
drops from first to last, and 256k extrapolates to ~80 s.

The allocation at the full 1.05 M-token pool moves to first, being the only
item that can fail outright rather than be slow, and bf8 KV accuracy at depth
moves to second because nothing measured so far tests it -- the 32k rung was
throughput, not quality. Concurrency-4 aggregate throughput is genuinely
unknown and now says so instead of being assumed fine: at ~80% fixed cost per
token, four streams might batch well or might contend.
Both remaining suspects for goal 3's ~54 ms fixed cost -- the 48 sequential
Gated DeltaNet state updates, and the per-layer all-reduce at TP=2 -- fit the
evidence equally well, and a device profile is the thorough way to separate
them. TP=1 on a single card is the cheap way: it removes every CCL round while
leaving the GDN chain untouched, and the weights fit, since down_proj at
bfloat4_b puts on-device resident weights at ~25.9 GiB against one card's 32.

Unchanged decode means the all-reduce is innocent and the work is in the GDN
kernel; substantially faster means TP=2's CCL is the cost and the TP=4 tuning
fallback recorded in TP2-AND-TT-FORGE.md becomes the target. Either answer
constrains what a profile has to explain, for one cold build.

Written down with the caveat that TP=1 is a diagnostic and not a candidate: it
halves available bandwidth and cannot hold goal 2's KV pool, so a good number
there is a finding about overhead, not a proposal to serve that way.
From the allocator's own numbers when 4 x 256k died in warmup: 8 banks of
4,142,923,648 B is 30.87 GiB per card, 61.73 GiB across the pair. The
spec-sheet 32 GiB is not all addressable, and the shortfall is bigger than the
margin this document's plans were built on.

It is exactly why 4 x 256k failed. The KV pool of 34.00 GiB allocated
successfully against 33.83 GiB of real headroom, left 179 MB, and warmup then
asked for 80 MiB it could not get -- with a largest free block of 10.2 MB, so
fragmentation as well as exhaustion.

Also records how to read that message, since it is the diagnostic for every
future device OOM on this host: bank size x 8 is capacity, allocated is what
is spoken for, and largest free block separates real exhaustion from
fragmentation.
Four concurrent 262,144-token requests allocated, served and came back
coherent. vLLM's own accounting: GPU KV cache size 1,048,832 tokens, maximum
concurrency 4.00x. That is goal 2's gate, functionally.

It needed BOTH memory levers together -- down_proj at bfloat4_b for 2.65 GiB
and the trace region halved for 1 GiB -- because usable DRAM is 30.87 GiB per
card rather than 32. With either one missing the warmup OOMs.

The decode numbers from that run are junk and the harness is why: prefill
SERIALISES. Stream 1 starts decoding while streams 2-4 are still prefilling,
so its inter-token gaps are other requests' prefill chunks, which reported
1.562 tok/s per stream against a true single-stream 14.7 and made the
aggregate look like a regression. With --gen 32 against ~2-minute prefills the
steady-state phase is a rounding error in the window. Documented in the
script: above concurrency 1, trust 'did it allocate', 'did it stay up' and
'is the text coherent', and use --gen 256+ for a rate.

The coherence is not an artefact, and it is the first evidence on the question
upstream flags with 'validate PCC at long ctx': the model read a ~240k-token
prompt and correctly described its structure with bf8 KV.
At 4 x 256k the four TTFTs came back 135.8, 270.8, 405.8 and 540.6 s -- exactly
135 s apart, because prefill serialises: one request at a time, ~135 s per
240k prompt. So each stream's own decode rate is mostly a record of how long it
sat interleaved with other requests' prefill chunks, and the four rates read
0.366, 0.537, 1.007 and 8.128 tok/s for four streams doing identical work.
Averaging those describes the queue, not the engine.

The bench now keeps absolute per-token timestamps and reports a steady-state
aggregate over [max(first token), min(last token)] -- the window in which every
stream was actually decoding. It prints that as the throughput number and says
so, with the per-stream averages demoted.

The last stream is the reason this is worth measuring rather than estimating.
It decoded at 8.128 tok/s with all four active, which implies ~32 tok/s
aggregate against 14.86 single-stream. If that holds it is a 2.2x batching win,
and it is exactly what goal 3's finding predicts: decode is ~80% fixed cost per
token, and a fixed cost is precisely what batching amortises. The corollary is
that the 36% MBU single-stream figure is not the machine's ceiling -- it is the
cost of running one stream at a time.
Measured over the 19.1 s window in which all four streams were decoding: 617
tokens, 32.242 tok/s aggregate, 8.06 per stream. Against 14.864 single-stream
that is a 2.17x batching win, and it moves MBU from 32.5% to 70.5% of the
measured 860 GB/s.

This is the same fact goal 3 found, read forwards. Goal 3 measured ~54 ms of
the 67 ms per token sitting outside the weight stream and concluded no dtype
policy could close an 11% single-stream gap. A per-token fixed cost is
precisely what batching amortises -- four streams share one weight read per
step -- so the two results are one result: 32.5% MBU is not the hardware's
ceiling, it is the cost of running one stream at a time.

Worth stating plainly because it changes what to optimise. The single-stream
number is the one that misses its gate; the number that matters for a serving
host is the aggregate, and that one is at 70.5% of DRAM.

Recorded with what the naive averages said -- 2.502 per stream, 10.006
aggregate -- because they look like a 4x concurrency regression and are purely
an artefact of prefill serialising at ~135 s per 240k prompt. The four TTFTs
came back 135 s apart to within a second.
The trace region is pure memory -- it bounds how many decode traces can be
held, not how fast they replay. At 1024 MiB per device the two regions are
2 GiB of the 61.73 GiB total, and the first 4x256k attempt died 0.17 GiB short
of its 34.00 GiB KV pool. Halving it is what made the run fit.

Leaving the default at 1024 and the passing configuration undescribed would
have made goal 2 unreproducible from this script, which is the thing the
script exists to prevent.
Operator, 2026-09-05: get a smaller quant running natively on a single card,
using the QUASAR NVFP4 build.

"Natively" is a claim about two axes at once, and only stating both makes the
goal falsifiable. A 20.6 GB NVFP4 checkpoint loaded under today's dtype policy
gives byte-for-byte the same device footprint as the 30.9 GB FP8 one, because
tt-metal re-tilizes everything on load and picks its own dtype. So the gate is
(1) the compressed-tensors weights are unpacked by our loader, no side
conversion, and (2) the device carries bf4 where QUASAR carries 4-bit.

The budget is derived from measurement rather than the spec sheet, and it is
close. Live device weights today sum to 25.853 GiB against 30.87 GiB usable per
card -- 5.02 GiB left, which will not serve long context. QUASAR's coverage
takes the four bf8 groups it quantised down to bf4 and saves 3.374 GiB, landing
at 22.479 GiB and freeing 8.39 GiB. Against 34,816 B/token of KV -- 16 full
attention layers, the 48 GDN layers hold a fixed-size state -- one 262,144
session needs 8.50 GiB, so it is 1.1 GiB short until tok_embeddings also drops
to bf8. That is a real lever held in reserve, not an assumption; goal 2 missed
its first attempt at this margin by 0.17 GiB.

Two corrections to standing documents, both from reading the artefacts:

- QUASAR does NOT quantise all Linear. Its config ignores lm_head. All three
  sources -- unsloth, QUASAR, and tt-metal's own policy -- independently keep
  lm_head at 8 bits, so that question is settled and should not cost an
  experiment.
- FP4-PRIORITY's ~29 tok/s ceiling used the 27.49 GiB figure, which counts
  prefill-only gate_up and the embedding table. The decode read is 20.17 GiB.
  More importantly goal 3 showed the ceiling is not the binding constraint at
  all: ~80% of per-token time is fixed cost, and moving down_proj to bf4 bought
  +3.1% where that section predicted +15%. The prize here is footprint, which
  is exactly what makes one card possible -- it is not the speed prize.

Expect single-stream decode well under the 28.3 tok/s roofline, around 10-14,
and note that this goal incidentally runs the TP=1 diagnostic goal 3 asked for:
if it comes in much faster, the fixed cost was the all-reduce, not the GDN
chain.

The goal4 serve profile is committed non-functional on purpose. The unpacker
and the coverage gate do not exist yet, but the mesh syntax does need
recording: MESH_DEVICE takes a literal tuple, "P300" is not in the plugin's
preset map, and a single visible card would otherwise resolve to (1, N) by
accident.
Unpacks one compressed-tensors Linear on CPU and checks it against the same
tensor from the FP8 checkpoint. Nine projections across MLP, GDN linear_attn
and full self_attn, first and last layers: PCC 0.994-0.996, norm ratio
0.984-0.998. That residual is the genuine difference between a 4-bit QAT build
and an 8-bit PTQ build of the same weights.

The format, now determined rather than assumed: two 4-bit E2M1 values per
uint8, LOW NIBBLE FIRST; per-group float8_e4m3 scales at group_size 16 along
the input dim; a scalar float32 weight_global_scale that DIVIDES. Nibble order
is not a guess -- high-nibble-first scores PCC -0.000410 on the same tensor.

Two checks, because one is not enough. PCC catches anything that scrambles
which value goes where. It cannot catch the global-scale direction: that
scalar divides uniformly, so inverting it leaves PCC at ~1.0 with every weight
wrong by a constant factor. The norm ratio is the only thing standing between
that error and the cards, which is why it is a gate and not a printout.

This runs off-device on purpose. A packing error produces a model that loads,
serves, and is subtly wrong -- and the accuracy gate would only find it after
a multi-hour cold cache build.

input_global_scale is deliberately not consumed: those are activation scales
and we quantise activations differently. Dropped explicitly rather than left
to fall through.
Adds is_nvfp4_checkpoint / unpack_nvfp4 / LazyNVFP4StateDict /
load_qwen36_state_dict_nvfp4 to weight_mapping.py, and a second branch in
model_config.load_state_dict. Applies to vllm-tt:src at --fuzz=0.

The result that matters is not "it loads" but that it loads into exactly the
same shape as the FP8 path: 851 internal keys on both, zero keys unique to
either, zero shape mismatches, zero dtype mismatches. Every TP loader
downstream of load_state_dict is therefore unchanged, and the checkpoint format
stops mattering at that boundary. Detection is mutually exclusive in both
directions.

Laziness holds, which was the thing most at risk. Peak RSS is 0.23 GB after
building the index over 2,687 tensors and 2.20 GB after reading five -- and
that 2.20 is the two 248320x5120 bf16 tensors materialising on demand, not the
projections, which were never touched. manifest() returns 851 meta entries. On
a 16 GB host with a 200 GB swapfile this is the difference between loading and
thrashing.

LazyNVFP4StateDict subclasses LazyFP8StateDict and keeps the _spec 5-tuple, so
substate/manifest/shape_dtype/iter/len are inherited rather than duplicated.
Only spec construction and __getitem__ differ.

Placed at the end of weight_mapping.py, after LazyFP8StateDict: the first
version of this patch inserted it before _remap_fp8_key and died at import with
NameError on the base class. The patch applies cleanly either way -- only
running it catches that, which is the argument for the import test being part
of the build and not a manual step.
layer.py branches on num_devices > 1, so a (1,1) mesh takes the NON-TP modules,
and those call load_gdn_weights, which raises "DeltaNet layer requires the
combined qkv_proj weight". Both lazy loaders deliberately keep GDN linear_attn.*
raw -- fused in_proj_qkv, fused conv1d -- because the TP loaders shard those
themselves. Goals 1-3 ran entirely at TP=2 and could not have found this; it
would have surfaced as a crash after a multi-hour cold cache build.

as_single_device() supplies the remap_qwen36_state_dict layout without giving
up laziness: in_proj_qkv renamed to qkv_proj, conv1d recorded as three slice
specs that slice on read. 851 keys become 947 (+96 = 48 GDN layers x 2 extra
conv keys), the NVFP4 and FP8 single-device key sets are identical, the split
reassembles to the original tensor exactly, and the spec shapes match the
actual reads -- which matters because the warm-cache marker trusts them. Peak
RSS 1.60 GB.

model_config passes single_device=(num_devices == 1), so two cards are
untouched.

I first read LINEAR_Q_DIM/LINEAR_K_DIM = 2048 as hardcoded for the 9B and
therefore wrong here. They are not: that is
linear_num_key_heads x linear_key_head_dim, 16 x 128 in both models, and V is
taken as the remainder -- 6144 here against the 9B's 4096. The stale constant
in that block is FULL_ATTENTION_LAYERS = {3..31}, and it is dead code.
Three findings, none of them in the original work list.

The NVFP4 format is now determined rather than assumed, with the nibble order
settled by measurement (high-first scores PCC -0.0004 where low-first scores
+0.9955). The loader is indistinguishable downstream -- 851 keys, zero
mismatches of any kind against the FP8 path -- which is the result that matters,
because it means nothing below load_state_dict has to know which checkpoint it
got. And a (1,1) mesh takes the non-TP modules, which goals 1-3 never reached;
that cost a patch the plan did not budget for and would otherwise have surfaced
as a crash after a multi-hour cache build.

The generalisation is worth more than the three facts: every patch this repo
carries targets the TP path, so the single-device path is unexercised and more
of it is likely to be wrong. Budget for that rather than treating each break as
a surprise.

Also records a correction I had to make while deriving it: LINEAR_Q_DIM = 2048
looks 9B-specific and is not.

The overlay Dockerfile grows 0009/0010 and an assertion that unpacks two bytes
and checks the E2M1 values, on the same principle as 0008's: a text check
proves the patch landed, only running it proves the behaviour.
G4.1: Qwen3.8-27B serves from the QUASAR NVFP4 checkpoint on a single p150a,
(1,1) mesh, one device, coherent output. Cold build 14 minutes, warm restart 8.

G4.2: device weights are 19.764 GiB against 30.87 usable, leaving 11.106 free
-- comfortably inside the 26.0 GiB gate. The goal doc predicted 25.853 and was
6.1 GiB pessimistic, because it derived the budget from the TP=2 cache: the
single-device layout carries no fused prefill mlp.gate_up.swiglu (5.98 GiB
there) and no fused qkvzab/wqkv duplicates. It is genuinely leaner, not merely
differently named, and every KV projection in that section should be re-derived
from 11.106 rather than 8.39. That is now enough for a full 262,144 session
with margin once bf8 KV works.

And this run is the TP=1 diagnostic goal 3 asked for. That doc predicted two
outcomes: unchanged, meaning the GDN chain is the cost, or substantially
faster, meaning the TP=2 all-reduce is. The answer is neither. One card is
2.14x SLOWER (6.94 tok/s, 144.1 ms/token, against 14.864 and 67.3), and MBU is
essentially IDENTICAL -- 30.1% against 32.5%.

So decode scales near-linearly with card count at constant MBU, and the CCL
costs about 7%. What goal 3 measured as ~80% "fixed cost" is fixed with respect
to WEIGHT BYTES, not with respect to hardware: it halves when you add a card.
It is per-card work -- compute or per-op dispatch -- not a serial latency floor
and not DRAM bandwidth. MBU was the wrong instrument for it.

That sharpens step D into a real prediction rather than a hope: if bf4 at
QUASAR's coverage buys much more than its byte share, the cost is matmul
fidelity (bf4 selects LoFi at 16 cycles/tile against bf8's HiFi2 at 32), which
is what QUANTIZATION.md argues and what down_proj alone was too small a share
to show.

One blocker found and worked around. QWEN_SDPA_BF8=1 dies in paged_fill_cache
with input_tensor.dtype() == cache_tensor.dtype(): the non-TP attention module
does not cast the prefill input to the cache dtype. The TP path does, which is
why goal 2 never saw it. KV_BF8 now gates it and defaults off on goal4, at the
cost of 2x KV memory -- a deferred fix, not a closed question.
The budget section was 6.1 GiB pessimistic because it derived from the TP=2
cache; the single-device layout is structurally leaner. The decode expectation
was a miss. And the TP=1 diagnostic answered outside both branches it was
designed around. Keeping the original derivations above and the corrections
below them, rather than editing the predictions to match, because the gap
between them is the part worth reading.
Asked whether there is an upstream PR for native FP4 so we can stop
dequantising to bf16. There is real MXFP4 work upstream -- #46838, #44277,
#42106, #51975 -- and every piece of it targets QUASAR, Tenstorrent's next-gen
architecture, not the QUASAR-QAT model. The names collide unhelpfully. grep
settles it: MxFp4 appears throughout tt_llk_quasar/ and in no file under
tt_llk_blackhole/. Blackhole's format enum has Bfp4 and Bfp4_b and no MX or NV
format at all, so a native NVFP4 matmul is not expressible on this silicon at
any effort, and there is nothing to wait for. Bfp4_b IS the native 4-bit format
here, and the MLP already lands on it -- 8.96 GiB of the measured 19.76 are
already 4-bit on device, not upscaled.

Then measured what fidelity and dtype are actually worth, because
QUANTIZATION.md's "BFP4 at LoFi is exactly 2x BFP8 at HiFi2" is a statement
about this hardware and we own some.

I went in with a hypothesis from tt-metal #39627 -- that LoFi consumes only
about 4 bits of the weight, so a bf8 weight would be truncated and bf4 coverage
would be free. That is refuted. bf8 at LoFi scores PCC 0.999876 against bf4's
0.993162; the bf8 mantissa bits do reach the engine. #39627 says LoFi consumes
the full BFP4 weight, which does not imply anything about BFP8. My prediction an
hour ago that bf4 might buy more than its byte share was wrong, and in the other
direction.

What the measurement does say:

The model already runs LoFi everywhere on the single-device path, so the LoFi
speed is banked and the 2x was never available. At fixed LoFi, bf4 over bf8 is
-28% on one decode shape and 0% on another with identical weight bytes -- the
reduction depth moves it, not the byte count. So bf4 coverage buys footprint,
not single-stream speed, and costs PCC 0.9932 vs 0.9999. Still worth doing on
one card, because footprint is what buys context and concurrency there, but it
has to be argued on those grounds.

And a free win pointing the other way: at decode shapes all three fidelities
take the same time (0.255 / 0.255 / 0.254 ms at M=32), which means the matrix
engine is idle waiting on weight movement and dispatch rather than doing math.
bf8 at HiFi4 is PCC 0.999971 against LoFi's 0.999876, so decode could run HiFi4
for nothing. Prefill could not -- there HiFi4 costs 1.84x.

That idle-engine result is the same conclusion the TP=1 experiment reached from
the other side, and the same mechanism goal 2's batching win exploited.
Operator pointed at Thatch-cloud/Tenstorrent.Blackhole-Qwen3.8-27B, which is
weeks ahead on this exact model and pushed today. Two things worth having.

First, independent confirmation. Their demo is 43.1 ms at B=1 and 45.3 ms at
B=8 -- eight times the work for five percent more time. That is the same
dispatch-bound conclusion our TP=1 experiment and our fidelity probe reached
from two other directions, and they quantify it: 68 small ops per layer in GDN,
11.0 ms/step at B=1. They are at 43.1 ms against our 67.3 on two cards, so
about 1.56x ahead.

Second, and worth more, their negative results. Speculative decoding measures
0.65-0.83x on the fused baseline -- slower. That was the obvious next idea for
a dispatch-bound decode and it is now closed without us spending the week.
Their lever J is bf4 read rate, not shipped, which corroborates our own
measurement that bf4 buys 0-28% by shape rather than 2x. Also closed: more
cores, packed gate|up, DRAM-sharded in-projections, CPU offload.

Their shipped levers are recorded with flags and gains. Lever C
(QWEN35_GDN_DECODE_BF16, QWEN35_GDN_STATE_BF16) is env-only and already exists
in our tree -- in gdn/tp.py, so it is inert on the single-device path. That is
the recurring shape of this goal: the optimised path is the TP path, and one
card gives up all of it.

So probed whether the TP path could run on a (1,1) mesh. It cannot as it
stands: all_gather asserts num_devices > 1 and reduce_scatter wants a
neighbour. Not a dead end -- both ops are the identity at TP=1 and the bypass
is mechanical, with most of the 51 CCL references funnelling through
tp_common.py. If it works we inherit lever C, the tuned program configs, our
own 0001-0003/0007, and every future graft, instead of porting each to a path
nobody else optimises.

Probing cost minutes. Finding it the other way costs a 15-minute cold build and
a crash mid-warmup.
Operator's call to refocus on two cards, matching the external project. First
lever off their ledger, and the cheapest: QWEN35_GDN_DECODE_BF16 holds the GDN
decode step at bf16 (high_precision=False on the recurrence op) and
QWEN35_GDN_STATE_BF16 holds the recurrent state at bf16 instead of fp32. Both
already exist in our gdn/tp.py -- they were inert only on the single-device
path, which is exactly why one card was the wrong place to be optimising.

Interleaved A/B in one session, same image and mounts, flag off then on --
their methodology rather than a stock-image control. 14.935 -> 15.359 tok/s,
66.957 -> 65.109 ms/token. Output stays coherent.

The control reproduces the goal-1 baseline of 14.864 to within 0.5%, which is
the check that this measures the flag and not the session.

Neither flag appears in build_variant(), so no cache variant and no cold build.
Their GSM8K for this lever is 57/60, equal to fp32; ours is unverified and
still waits on the accuracy baseline.
Operator's call: refocus on two cards, where the external project is and where
every optimisation actually lives. Lever C landed already (+2.8%, env only).
Lever A is the big one, +15.5% on its own, and it is a new C++ ttnn op so it
needs the wheel rebuilt.

Rather than guess at a 20-minute build, dry-ran the PR against our fork first.
It does not apply, and the reason is informative: our fork cfx/qwen36-p300-tp2
already carries FOUR of the PR's files -- text_demo.py, gdn/tp.py, tp_common.py
and conv2d.cpp all report "previously applied", which says the fork shares
#53587's lineage. What it does NOT carry is the op itself: no
ttnn.transformer.decode_gated_delta_rule binding, no source directory, no
QWEN_GDN_FUSED_DECODE anywhere in models/. So the rebuild is genuinely
required, and the rebase is two hunks rather than twenty-three files.

Both conflicts are against edits we already understand. ttnn_gated_deltanet.py
@217 collides with our own carried FIR-tap fix -- the (B, k+T, D) one-liner the
Dockerfile asserts and seds -- and that file is where the wiring lives, so it is
the hunk that matters. slice.cpp @325 collides with #53319, which the fork
carries. The 15 new op files apply clean, being new.

Filed under patches/upstream/ rather than patches/tt-metal/, because everything
in the latter is expected to apply at --fuzz=0 and is globbed by a Dockerfile.
That glob already shipped one broken image; a non-applying patch sitting next to
the applying ones is how it happens again.

Recorded one more trap for whoever wires it: Dockerfile.vllm-tt-src exports the
source with git archive HEAD, which does not see an unstaged git apply. Patch in
the wheel stage, commit before the archive, and assert the binding exists in the
final stage -- otherwise the wheel gets the op and /opt/tt-metal-src ships Python
that never calls it.
The rebase I mapped last commit turned out not to need resolving. Six of the
PR's 23 files are already in the fork -- it shares the PR's lineage -- and the
two that would have conflicted (ttnn_gated_deltanet.py against our carried
(B, k+T, D) FIR fix, slice.cpp against #53319) are not needed at all. The op
is wired through ttnn_delta_rule_ops.py, and gdn/tp.py:1094 already calls it.
Dropping the six leaves 17 files that git apply --check takes with no
conflicts, verified inside vllm-tt:src.

That reading came from the external project's own graft Dockerfile, which
copies exactly one Python file next to the rebuilt .so. It named the seam.

The build wiring commits the graft before 'git archive HEAD' so the wheel and
the /opt/tt-metal-src overlay cannot disagree, exports the device kernels
(tt-metal JIT-compiles them at run time, so the .so alone is not enough), and
asserts the symbol in the .so and the binding in Python -- neither of which
'it built' would prove.
We have optimised single-stream decode for three goals and never once measured
what the pair does under load. It does 5.53x: 14.358 tok/s at one stream,
79.452 aggregate at eight, for a 31% per-stream cost. That ratio reproduces
the external project's 5.5x on the same shape.

This is the number that answers the operator's standing question about
leveraging the combined 860 GB/s, and it was a config change, not a build.
QWEN_BATCHED_GROUPED=0 is load-bearing above B=4 with short prompts: grouped
batched prefill asserts BH <= ncores, and TP=2 with 48 value heads gives
8 x 24 = 192 against 110 compute cores. goal3 now takes SEQS/CTX and sets the
flag itself rather than leaving it to be rediscovered.

Also recorded: lever E is not available to us. I had found the num_links=1 at
tp_common.py:514 independently and had an A/B running; their plan says that
function has no call sites, our tree agrees, and the run was killed before it
spent twenty minutes measuring an uncalled function. The real link lever needs
a second QSFP-DD cable, which is an operator decision, not a patch.
Its optimisation work is on an unmerged branch -- 14,005 lines: a 2,284-line
measured plan, eight lever patches, three custom TTNN ops with kernels, and
the graft Dockerfiles. main is documentation and CI, which is why the earlier
pass only found a results table.

Six of their levers are measured NEGATIVE, and one of those is speculative
decoding at 0.65-0.83x -- the obvious answer to a dispatch-bound decode, and
weeks we now do not spend. Three more are closed at roughly zero, including
matmul core count, which matters because their own patch exposes that
_find_grid caps Blackhole's 110 cores at a Wormhole 8x8. That reads like a
free 3x until you find their number for it: 0.3 ms.

Their v0.77 is not our v0.79, so every claim that depends on dead code or a
hardcoded table was re-checked here rather than inherited. Two changed: lever
E is a second-cable hardware ask for us and not a patch at all, and our fabric
needs no TT_MESH_GRAPH_DESC_PATH at this version.
Six arms, two interleaved B=1 pairs and a B=8 pair, same image throughout --
the control is the graft with the flag off, never a different image, so the
only difference between arms is the code path taken. Engagement asserted in
both directions: every treatment arm logged the fused op's one-shot line and
every control arm logged none. A control that had silently engaged would have
reported the treatment twice and read as a null result.

B=1  15.539 -> 15.898 tok/s  (+2.31%, -1.45 ms/token)
B=8  89.608 -> 98.100 tok/s aggregate  (+9.48%, -7.73 ms/token)

The external project claims +4.7% and +12.9%. We get about half their B=1
figure and three quarters of their B=8 one, but the SHAPE reproduces exactly
and that is the part that matters: the fusion is worth five times more in
absolute milliseconds at B=8 than at B=1, because the composed small ops stop
being floor-bound once each carries eight rows while the fused op amortises
them. That is the same mechanism their profile describes, arriving at our
numbers independently.

Recorded honestly rather than rounded up: our arm-to-arm spread is 0.5-0.9%
against their 0.1%, so the B=1 result is above noise but not far above it.
The B=8 result is well clear. B=1 TTFT rose with the flag, which is first-call
JIT of the new kernel landing inside the measured request -- prefill does not
take this path -- and is not chased here.
The two-card decode deficit is now arithmetic rather than suspicion. Two cards
carry ~430 GB/s each and one decode step reads the whole 27B model once, so the
bandwidth floor for a token is about 36 ms. We are at 63 ms. Over half of every
step is not weight traffic, and the external profile on identical silicon names
what it is: 68 small ops per layer across the 48 GDN layers, 97% device-side.

So the goal tests one thesis to destruction -- that the deficit is op-launch
overhead inside GDN and closes by fusing, not by quantising further, not by a
faster interconnect, and not by speculation. The last three are already
measured negative on this hardware; the doc names them so they do not get
reopened.

The gate is 18 tok/s single-stream, 115 aggregate at 8 streams, GSM8K >= 55/60,
and an A/B with engagement asserted in both directions for every shipped lever.

Honest projection up front: applying our own lever-A haircut (half their B=1,
three quarters of their B=8) to K + 3c + M gives ~17 tok/s, which is short of
the title. Section 4 says where the last tok/s would have to come from and
admits both candidates may come up empty -- including a planned re-measure of
lever H after K, recorded so that rediscovering the 8x8 Wormhole grid
assumption produces a step rather than a second round of excitement.

Order is 3c first as a near-zero-risk rehearsal of the graft-assert-A/B loop,
then K in its four sub-flags rather than as one 1,600-line graft, then M, then
the two profile levers.
Lever A's graft of upstream #53587 already carries `inplace_state` the whole
way -- nanobind signature, op attributes, C++ writer. Nothing ever passed True.
So 3c is two call sites, 86 lines, and no ttnn rebuild: the overlay builds in
three seconds against vllm-tt:leverA4 where the src Dockerfile is an overnight
job on the fork's compile stage.

That also means 3c needs no second image on cfx-llm2. The flag is env-gated
inside one image, so the control is that image with the flag off -- which is
the control discipline we already use, and it takes the disk pressure out of
this step entirely.

Three build assertions rather than a grep, because 3c can fail in three ways
that all look identical from a benchmark: the wrapper not forwarding the
kwarg, the grafted C++ op predating it (a silent no-op if the base were an
older revision of the PR), and tp.py gating on env alone. That last one is not
cosmetic -- in-place is only legal when init_state IS the persistent buffer, so
an env-only gate would ask for it on a sliced state and corrupt decode at
B < Bmax. The patch guards on identity and the assertion checks for the guard.

The buffer_address() check in the patch is what makes the A/B honest: nanobind
returns a NEW Python wrapper around the same device buffer, so `is` sees two
objects and a lever that silently fell back to the copy path would look like a
clean null result.

Not added to Dockerfile.vllm-tt-src yet -- that is the canonical from-nothing
build and 3c has not been measured. It goes in if the number holds.

Theirs: -0.42 ms at B=1, -3.26 ms at B=8.
Lever A's driver has been living in /tmp on cfx-llm2, which is exactly where a
measurement harness should not live: the numbers in bench/runs are only
reproducible if the thing that produced them is in the repo next to them.

3c's driver differs from A's in one way that matters. A's engagement check is a
single grep for one log line; 3c needs two, because 'requested' and 'happened'
are different facts. The env var can be read and the identity guard can pass
while the op still falls back to the copy path -- and since nanobind returns a
new Python wrapper around the same device buffer, the fallback is invisible to
an `is` check. The patch prints happened=True/False from buffer_address(), and
the driver treats a treatment arm that requested in-place without getting it as
a second control rather than as a result.

Also two passes at B=1 rather than one, and a note in the script saying why the
second may still not settle it: 3c's claimed B=1 gain is 0.42 ms, which is
smaller than the 0.5-0.9% arm-to-arm spread lever A's run actually showed. If
the arms interleave, that is the finding.
The goal doc named this as K's largest risk: their ops are written against
tt-metal 9f9cd4fd (v0.77.0-rc1) and we are on v0.79.0-dev20260903, with no
upstream PR to smooth the gap the way lever A had. Their own Dockerfile.k warns
the .so is ABI-bound to that rev, and their register_ops.py notes the
transformer ops are unity-built so per-file `namespace cb` blocks collide.

Grafted gdn_conv_gates -- the first of K's four sub-steps -- into our fork's
tree and linked it. It compiled clean on the first attempt, no API drift and no
CB collision, and `ttnn::prim::gdn_conv_gates` is in _ttnncpp.so. One of the
four, so this is not proof the rest follow; but the failure mode the doc
budgeted real rebase work for did not appear where it was most likely to.

One adaptation was needed: register_ops.py anchors its CMakeLists kernel glob
on `gdn_decay/device/kernels/*.cpp`, an op they carry and we do not. Ours
anchors on decode_gated_delta_rule instead. The other three anchors -- both
sources.cmake entries and both nanobind sites -- matched our tree unchanged,
which they only can because we already grafted #53587.

scripts/k-build.sh is the loop that made this affordable. `--target wheel`
tags the src Dockerfile's compile stage, which still carries
/tt-metal/build_Release, and a container off it re-links incrementally: 0.4 s
for a no-op, ~90 s for a fresh op through the transformer unity build, against
~20 minutes for the full image. That matters for more than convenience -- with
a 20-minute cycle the temptation is to graft all four sub-steps at once, and a
combined graft that comes out 3% slow tells you nothing about which quarter was
wrong.

The graft step replaces rather than nests, and syms asserts on nm output,
because an op missing its sources.cmake entry links perfectly and simply is not
there.
gdn_norm_gate and attn_prep link as cleanly as gdn_conv_gates did, so all four
ops (those three plus the grafted decode_gated_delta_rule) are in _ttnncpp.so.
The version gap the goal budgeted real rebase work for is not there.

That moves K's risk entirely into Python: patch_packed.py against the grafted
op, and the wire-k*.sh edits against our tp.py. Those use assert count == 1 on
their anchors, so they fail loudly instead of fuzzing -- which is the failure
mode worth having.

Note this closes M's build risk too, since attn_prep is M's op.
QWEN_GDN_CONV_GATES collapses the GDN conv shift-register, FIR, SiLU and both
gates into one device op -- twelve ops per layer across 48 layers.

Built as an overlay the way the external project builds theirs, not as a full
image rebuild, and the reason that is legitimate is worth stating rather than
assuming: vllm-tt:builder is --target wheel off our own
Dockerfile.vllm-tt-src, and that stage was a CACHE HIT against the build that
produced leverA4. Same fork, same A graft, same compiler, so the .so is
ABI-compatible with the wheel already installed in the base. If the wheel stage
is ever rebuilt cold while the base is not, that breaks -- as a runtime symbol
error, not a build failure, which is precisely what the import assertion is for.

Two traps, both hit on the first attempt and both now documented in the file:

The RUNPATHs differ. The wheel's extension resolves _ttnncpp.so relatively
(/build/lib:...), the build tree's absolutely
(/tt-metal/build_Release/ttnn:...) into a directory this image does not have.
Copied in unmodified it gives ImportError at the first . patchelf
rewrites it to the wheel's; without that the overlay approach does not work at
all.

And the names differ: _ttnn.cpython-310-x86_64-linux-gnu.so in the wheel,
_ttnn.so in the build tree.

The Python wiring is the external project's own wire-k.sh milestone-1 payload
run unmodified against our tp.py -- checked in as patches/k/wire-k1.py with its
output, so the produced file is reproducible rather than a copied artifact.
It applied clean, and the binding name it gates on
(ttnn.transformer.gdn_decode_conv_gates) matches what the nanobind actually
registers. That check matters more than it looks: tp.py gates on hasattr, so a
renamed binding fails CLOSED and the A/B would read as a clean null result
rather than an error.

Not measured yet. This is a build, not a number.
The 3c run in progress measured a control-to-control spread of 1.2% at B=1 on
this host (15.886 vs 15.697 on identical arms), which is wider than 3c's whole
effect. Two passes cannot separate a sub-1% lever here, so K1 gets three -- and
if three still interleave, that is the finding rather than a direction.

Both arms carry A and C, because K1 replaces part of the path A already fused
and a base without A would credit K1 with A's win. 3c stays OFF in both arms
even though the image supports it: K1 is the variable, and stacking an
unsettled lever underneath would make a null result unattributable.

The engagement check is FATAL on a treatment arm at zero rather than reported
as 0%, because tp.py gates on hasattr() -- a missing or renamed binding fails
CLOSED, producing no error, no speedup, and an arm indistinguishable from a
control.
patch_packed.py rewrites eight files of decode_gated_delta_rule so q/k/v come
in as ONE [1,B,C] tile tensor -- the conv+gates kernel's own output, channels
[q|k|v] head-major -- with the reader gathering head (b,h)'s rows straight out
of that layout. The model's slices, reshapes and repeat_interleaves for GQA go
away.

It applied to our tree unmodified: all eight files, all thirteen of its
asserts, and the result links. That is the second of K's four sub-steps whose
C++ needs no rebase, after milestone 1.

Worth being precise about why this was ever in doubt and why it is not a
surprise in hindsight: our decode_gated_delta_rule is the 17-file cut of
upstream #53587 and theirs is the same PR, so patch_packed is being applied to
the tree it was written against. The version gap between v0.77 and v0.79 lives
in tt-metal around the op, not in the op.

Checked in as the source patch rather than the patched tree, so it stays
reproducible against whatever #53587 cut we carry.

Still no measurement -- milestone 2 needs its Python half (wire-k2.sh) and the
model-side packed entry before it can be run, and milestone 1 is the one with
an image.
Six arms, engagement proven both ways on every one -- and for 3c that needed
two counts, not one. Every treatment arm logged requested=48 happened_true=48:
all 48 GDN layers asked the fused op to write through, and all 48 got it. Every
control logged 0/0. A treatment that had quietly fallen back to the copy path
would have shown requested=48 happened_true=0 and looked like a null result,
because nanobind returns a new Python wrapper around the same device buffer and
an `is` check cannot see the difference.

  B=8 per-stream  12.329 -> 12.660   +2.68%   -2.12 ms
  B=8 aggregate   98.636 -> 101.279  +2.68%
  B=1 mean        15.791 -> 16.177   +2.44%   -1.51 ms

Adopted. But the two rows do not deserve equal confidence and the record says
so.

B=8 is one pair, and it lands at 0.65x their -3.26 ms -- close to the 0.75x
lever A came in at on the same arm, which is the kind of agreement that makes a
single pair believable.

B=1 is not quantifiable at this sample size, and the honest tell is that it
looks too good. The mean implies 3.59x their -0.42 ms, which is not credible
for a lever whose whole mechanism is one avoided copy. The scatter explains it:
the two controls differ by 1.20% and the two treatments by 2.52%, both larger
than the effect being measured. What survives is the direction -- all four B=1
arms rank treatment above control, and the second pass swung +4.34% -- so the
lever is not harmful and is very likely positive. The number is not evidence.

The driver already anticipated this and said so in a comment before the run;
the K1 driver it seeded takes three B=1 passes instead of two.

Two cards now: 16.18 tok/s single-stream, 101.3 aggregate at 8.
Reading their wire scripts suggests K is copy-and-build. Auditing every path
they reference says otherwise: wire-k2.sh and wire-k3.sh both start by copying
~/kwork/ttnn_delta_rule_ops.py, the packed-capable Python wrapper, and that
file is not in the repository. Everything else -- all four op trees, every
patch script, register_ops.py -- is there.

So K milestones 2 and up cannot be reproduced by copying. Milestone 1 is
unaffected, which is lucky rather than planned: gdn_conv_gates wires straight
into tp.py and needs no wrapper change, so the sub-step we already built and
are measuring right now is the one that did not need the missing file.

The gap is small and the ledger says so rather than just flagging it: the
packed op's nanobind signature is public via patch_packed.py, and the entry
wire-k2 calls maps onto it nearly one to one -- about thirty lines mirroring
the _decode_gated_delta_rule_fused helper #53587 already gives us.

Two other things the audit turned up. patch_state_fast.py is labelled NOT
ADOPTED in its own docstring, a measured negative kept as a record -- not a
lever. And patch_reader_fast.py has no env gate at all: it is kernel-only, so
it needs no Python, but being ungated it cannot be A/B'd inside one image. That
is affordable now only because the data-root move lets cfx-llm2 hold a control
and a treatment image simultaneously -- the first lever whose measurement
depends on that change.
patch_reader_fast looked like a free-standing win: no env gate, and it touches
only two kernel files, both JIT-compiled from /opt/tt-metal-src at run time. No
ttnn rebuild at all -- a seconds-long source overlay like 3c.

It does not apply to our tree. It fails on 'gather_scalar zero anchor', which
is text patch_fuse_ng.py introduces, exactly as its own docstring says. Tested
rather than assumed, because the alternative was to schedule it as a cheap
lever and find out during a build.

Charting the rest the same way: only milestone 1 is standalone, and K2 through
K5 plus reader_fast all sit behind the packed path, which sits behind the one
file they did not publish. So the critical path for four fifths of K's claimed
value is about thirty lines of Python wrapper -- not a rebase, not a kernel
rewrite. Worth knowing as a shape rather than discovering one failed patch at a
time.

Nothing here changes K1, which is on the card now.
Two A/Bs in a row produced B=1 arms scattering wider than the effect: 3c's
controls 1.2% apart and treatments 2.5%, and K1's second control landing ABOVE
its first treatment. I was treating that as this host being noisy.

It is more specific than that. Their milestone-1 section reports -1.78 ms at
B=1 from exec_sync -- device step time -- and prints the tok/s column beside it
only to dismiss it: 'the tok/s column is host mode's readback noise (4.9-7.1 ms
between arms); exec_sync is the number, as everywhere else in this plan.' Their
tok/s arms read 19.23 / 19.54 / 19.50 / 19.01 and interleave just like ours.

Our harness measures endpoint tok/s. So our B=1 arms are not weak evidence
about these levers, they are evidence about readback jitter, and adding passes
cannot fix a systematic error of that size. The K1 driver's three-pass design
was the wrong remedy for the right observation.

What follows: B=8 is the decisive arm for us -- which is also what their own
sequence says, 'measured at B=8 first' -- and it has behaved cleanly here
(lever A +9.5%, 3c +2.68%) because eight streams amortise the jitter that
dominates a single stream. And a device-time measurement is now the
highest-value improvement to the harness, not the next lever.

Recording their milestone-1 B=8 figure to compare against: -2.40 ms (-4.1%),
independently corroborated by a device microbench at -2.27 ms.
Three A/Bs in and the B=1 arms have never once separated: 3c's controls 1.2%
apart, K1's second control above its first treatment. I read that as a noisy
host and answered it with more passes. It is not a noisy host, it is the wrong
instrument, and their plan says so beside its own numbers -- 'the tok/s column
is host mode's readback noise (4.9-7.1 ms between arms); exec_sync is the
number, as everywhere else in this plan.' Their tok/s arms interleave exactly
like ours.

exec_sync is the host-side time around the traced device step, with update and
readback timed separately so only the model is counted, and their cross-check
puts it within 0.24 ms of DEVICE TRACE FIRMWARE/KERNEL DURATION.

The good news is that it needs no new code. text_demo.py in our own fork
already instruments it, behind QWEN36_DEBUG_DECODE_TIMING=1, and ships the
same two config ids every result in their plan is quoted against: traced_128
for B=1, batched_128_b8 for B=8. This driver is the fifty lines that read it.

It runs the demo as a pytest to completion rather than serving an endpoint --
no readiness poll, no HTTP, no readback path -- which is the whole point.

Engagement discipline is unchanged, plus one addition the endpoint drivers did
not need: a missing exec_sync line is FATAL rather than an empty result, since
a demo that died or an unhonoured flag would otherwise report as a blank arm.
The operator asked for the mental model rather than the status: what a Tensix
core is, how a model gets onto two Blackholes, why decode behaves the way it
does, and what we are actually tweaking. Written so future sessions can be
steered on mechanism instead of on lever names.

The organising idea is section 5, and it is the one that explains every
decision in this programme: decode is NOT bandwidth-starved. The bytes say the
floor is ~26-28 ms/token at TP=2; the step is ~55 ms and 97% of it is device
time; and inside that, 11 ms/step is 3,264 GDN op invocations averaging 3.38 us
each regardless of how little data they touch, with 60 of the 68 ops per layer
under 5 us. It is a per-op floor. So fusion is the whole strategy, and
speculative decoding -- the textbook answer to a memory-bound decode -- measured
0.65-0.83x, slower, precisely because the premise does not hold here.

Grounded rather than recalled. The Tensix internals are quoted from tt-metal's
own METALIUM_GUIDE.md in the build container; the JIT claim is checked against
tt_metal/jit_build/ and kernel.cpp's includes; the unity-build collision
against the CMake flag and our op's own namespace cb block; the model dims
against the checkpoint config.json on cfx-llm2; core count, RAM and topology
against the host; GDN's decode path against the real forward_decode in tp.py;
exec_sync against the timing block in text_demo.py. Everything from the
external project is attributed as theirs.

Two corrections to our own docs fell out of writing it. GOAL-KERNEL-LEVERS.md
estimates the bandwidth floor at ~36 ms from ~30 GiB of weights; that
over-counts, because it uses device footprint rather than bytes actually
streamed and the MLP gate/up tensors are bf4. The better-grounded floor is
~26-28 ms -- which strengthens the argument rather than weakening it, since the
gap to reality is larger than we wrote. And the 25.853 GiB weight figure is the
single-card NVFP4 budget and must not be read as a TP=2 per-card number.

The closing section lists what is theirs versus ours and what I could not
verify -- notably that METALIUM_GUIDE's NoC diagram is labelled Wormhole, so
Blackhole's per-core L1 size is not independently confirmed here, and that the
GDN state equation is an intuitive rendering rather than a transcription of the
compute kernel.
Section 1 derived a ~36 ms floor from 'roughly 30 GiB of weights'. That is the
device footprint, not the bytes streamed per decode step, and it also ignores
that the MLP gate/up tensors are bfloat4_b rather than bf8, so fewer bytes move
than the footprint implies. Counting streamed bytes gives ~26-28 ms, a ceiling
nearer 36-38 tok/s than 28.

Surfaced by the HOW-IT-WORKS.md pass, which did the byte table properly rather
than reusing my round number.

Worth being clear that the correction is not flattering: a lower floor means
the gap between the floor and our measured 63 ms is LARGER, so more of every
step is unexplained by weight traffic. It strengthens the dispatch-bound thesis
this goal exists to test. The projection in section 4 is untouched -- the
bandwidth ceiling was never the binding constraint, which is rather the point.
Eight arms. Every treatment logged engaged_lines=48, so all 48 GDN layers took
the fused conv+gates path; every control logged 0.

  B=8 per-stream  12.289 -> 12.757   +3.81%   -2.99 ms
  B=8 aggregate   98.316 -> 102.058  +3.81%

Theirs is -4.1% / -2.40 ms, independently corroborated by a device microbench
at -2.27 ms. We reproduce 93% of their FRACTION. That is much closer than
lever A (which came in at half their B=1 and three quarters of their B=8), and
the reason to trust it is that their two independent methods agree with each
other before ours agrees with either.

Our absolute saving is LARGER than theirs -- 2.99 ms against 2.40 -- while the
fraction is slightly smaller, because our B=8 step is 81.4 ms where theirs is
57.9. Removing 12 ops x 48 layers should be a roughly constant number of
microseconds, not a constant fraction, so a bigger absolute saving on a slower
step is the arithmetic behaving as expected. A tempting reading is that
op-launch costs more on this host and so fusion pays more here; that is a
hypothesis, not a result, and their own profile puts host round-trip at only
1.76 ms of 56.21, so most dispatch cost is device-side and the host CPU is
probably not the explanation.

B=1 is recorded as NOT MEASURABLE rather than as +0.47%. The three passes read
+0.62%, -0.20%, +1.01% against a 2.14% control spread, and we now know why:
endpoint tok/s at B=1 is the readback noise their plan explicitly sets aside.
Their B=1 figure for this lever is -1.78 ms from exec_sync; ours is not
comparable and is not reported as if it were.

Adopted. Two cards: 102.1 tok/s aggregate at 8 streams.
One place for the best measured numbers, so a future session does not have to
reconstruct them from four bench/runs entries and three commit messages.

Admission rule stated at the top: a number enters only if its A/B asserted
engagement in both directions. That keeps projections, borrowed figures and
unengaged arms out by construction.

The single-stream record is marked as the best single ARM rather than a mean,
with the honest figure (~16.0 +/- 0.2) beside it, because our B=1 arms scatter
1.2-2.5% on readback jitter. Publishing 16.38 without that caveat would be
choosing the luckiest arm and calling it a record.

Also records what today's progression was NOT: the 14.36 -> 79.45 jump at 8
streams was not a lever, it was measuring batched serving for the first time on
a host that could always do it. The honest lever total at B=8 is 79.45 ->
102.06, +28.5%.

Prefill is listed as in progress rather than omitted, because its absence is
itself worth seeing: every number in this programme so far is a decode number.
Every number this programme had produced was a decode number. This is the other
half of the serving story, swept 128 -> 32,768 tokens on the full adopted stack
with engagement asserted before any number was read.

  128     0.80 s     160 tok/s
  512     0.54 s     950
  2,048   0.64 s   3,190
  8,192   2.34 s   3,502
  16,384  4.56 s   3,605   <- peak, and the cleanest point
  32,768  9.76 s   3,356

The curve matters more than the peak.

Short prompts are latency, not throughput: at 128 tokens roughly 0.77 s of the
request is engine scheduling, HTTP and one decode step rather than prefill
arithmetic. Reporting 160 tok/s as 'prefill speed' would be reporting our own
fixed overhead. By 16k that overhead has amortised to ~4 ms, which is why that
point is the honest one.

32k comes in 6.9% below the plateau, which is the 16 full-attention layers
starting to cost quadratically -- and is precisely the regime lever D
(QWEN_SDPA_BF8) exists for and where our stack deliberately does not yet enable
it.

One thing this sweep does NOT establish, said plainly in RECORDS: it was taken
with the levers ON, so it is a baseline, not evidence that the decode fusions
are neutral for prefill. The theory says they should be -- prefill is
compute-bound where decode is op-floor-bound -- but a flags-off prefill arm
would be the evidence and has not been run.
Answers the operator's question about chip-level headroom and whether any part
of the model belongs in SRAM, with arithmetic rather than adjectives.

Establishes the hardware constants from source rather than from a diagram.
Blackhole's per-core L1 is 1,572,864 B, read from the Blackhole-specific
dev_mem_map.h and corroborated by the SoC descriptor -- which retires the
'unverified, taken from a Wormhole-labelled diagram' caveat HOW-IT-WORKS.md
carried. 39 KiB of that is system-reserved, per the header's own comment. And
110 cores is now structural rather than merely measured: our cards are
2xharvested, whose compute_with_storage_grid_range is [0,0]-[10,9] = 11x10,
with a further column of 10 Tensix reserved for dispatch.

Total SRAM is 165 MiB per card, which is 1.9% of the 8.69 GiB the model streams
per card per decode step. That ratio is what makes the SRAM question tractable:
weights are never a candidate, so it reduces to the small persistent tensors.

The central finding is a saturation one. Counting streamed bytes at the dtypes
actually used -- bf8 projections, bf4 MLP gate/up/down -- the per-card floor is
21.71 ms against a measured 61.8 ms step at B=1. So 65% of a decode step is
neither weight traffic nor arithmetic, effective bandwidth is 151 GB/s of 430,
and achieved compute is 0.084% of ~491 TFLOP/s with an arithmetic intensity of
2.7 FLOP/byte against a 1,141 FLOP/byte roofline knee. This corrects
GOAL-KERNEL-LEVERS.md's floor a second time, again in the direction that makes
the unexplained gap larger.

On the SRAM question the answer is a qualified yes with an awkward shape. The
GDN recurrent state (B, 24, 128, 128) per card is DRAM-resident today --
reset_state() allocates it with no memory_config -- and is the only tensor that
is small, persistent, and touched every step. But it fits in L1 at B=1, where
it is worth 0.18 ms, and needs 288 MiB at B=8, where it would be worth 1.40 ms.
Partial residency at batch is the realistic form and lands around 0.4-0.6 ms,
which ranks below the packed-path work.

The byte model is validated rather than asserted: it predicts lever 3c's gain
at 1.72 ms against our measured 2.12 ms with no fitting, and it corrects an
assumption I had been carrying -- roughly 80% of 3c's B=8 win was bytes, not op
launches.

Also derives the remaining op-launch budget after A + C + 3c + K1: about 41 of
the original 71 ops per GDN layer, worth ~6.65 ms at B=1 and ~12.79 ms at B=8,
essentially all of it addressed by K2-K5. That is the ranked table's top three
rows and they share one blocker, the unpublished packed wrapper.

Two things I will not overclaim. Lever H's 'launch-bound' framing looks wrong
-- decode matmuls are DRAM-bound, and the real question is whether 33 cores can
issue enough outstanding NoC reads to saturate 430 GB/s, which their pre-fusion
measurement could not have answered. And the 3.38 us per-op average is
demonstrably not uniform, since lever M's measured -0.50 ms is 3x below what it
predicts, so the remaining-dispatch figure is an upper bound.

Unverified items are listed in a closing section, including the runtime L1
allocator base, the attn_output_gate projection shape, and the fact that every
per-op and per-lever constant in the ranked table is theirs rather than ours.
Second correction to the same paragraph in one day, subtler than the first. The
26-28 ms figure came from the external project's byte table, which lists MLP
down at bf8. Ours is bf4 -- QWEN36_MLP_DOWN_BF4=1 is in the goal3 profile every
benchmark today ran under, verified in scripts/vllm-tt-serve.sh.

Recounting at the dtypes we actually serve: 8.692 GiB per card per step, floor
21.7 ms, ceiling near 46 tok/s. The two figures are not two analyses
disagreeing; they are two stacks, and the difference is one dtype. Both are
kept, each next to the stack it describes.

The gap widens again. At ~62 ms measured, effective bandwidth is 151 GB/s of
430 (35%) and achieved compute is 0.084% of peak -- roughly two thirds of a
decode step is neither weight traffic nor arithmetic.

Also checks in bench/ab/prefill-sweep.sh, which produced today's prefill curve
and was sitting untracked.
A record without a utilisation figure invites the wrong follow-up question. At
16 tok/s the interesting fact is not the number, it is that producing it uses
35% of DRAM bandwidth and 0.084% of the FLOPs -- so the next idea should be
about op count, not about quantisation or clocks.

Carries the caveat that matters for planning: the 3.38 us per-op floor is an
average and is demonstrably non-uniform, since lever M's measured -0.50 ms is
3x below what the average predicts for its 30 ops. The ~6.65 ms of remaining
GDN dispatch is an upper bound, not a forecast.
The operator asked whether further work belongs in tt-metal, TTNN, or higher.
The answer is specific and not the intuitive one, so it is worth a document
rather than a paragraph: almost all remaining headroom is at the TTNN
op-granularity layer, the boundary between the model's Python and the device
operations it issues.

Layer by layer, with the reason each is or is not worth attacking. The
hardware is not the constraint (0.084% FLOP, 35% bandwidth). tt-metal's per-op
floor is not ours to lower, only to amortise. Quantisation is actively
counterproductive for speed because smaller tiles read slower per byte. Serving
is a large multiplier but a different kind of win -- more tokens from the same
steps, not faster steps.

Two things the document is careful about.

The model's Python is not a layer of headroom, it is the authoring surface for
one: host round-trip is 3% of the step, so a faster tp.py is worth nothing
while a tp.py that issues fewer ops is worth everything.

And 3c is written up as the instructive exception rather than smoothed over: it
removes no operations at all, and ~80% of its win was bytes. So the thesis is
'fewer ops AND less data those ops drag', not 'fewer ops'. Getting that wrong
would mis-rank the SRAM work, which is exactly the kind of idea the pure
op-count reading would dismiss.

Ends with the forward order and a four-question test for placing a new proposal
on the map before costing it.
This is the ~30 lines that everything past K1 was waiting on. wire-k2.sh and
wire-k3.sh both open by copying ~/kwork/ttnn_delta_rule_ops.py, and that file is
not in their repository -- audited this morning, and it is the only input of
theirs that is missing. So it becomes ours to write.

The C++ half stays theirs: patch_packed.py, already verified applying to our
#53587 cut and linking. This is only the Python entry that reaches it, mirroring
_decode_gated_delta_rule_fused which #53587 gives us.

Verified rather than assumed, in three ways.

Every keyword the wrapper passes was checked against the actual nanobind
signature in the built op -- qkv, beta, g, num_k_heads, num_v_heads, head_k,
head_v, scale, initial_state, inplace_state, memory_config. That check is not
pedantry: tp.py gates on hasattr, so a wrong name fails CLOSED, producing no
error, no speedup, and an arm indistinguishable from a control.

The signature was checked against how wire-k2 and wire-k3 actually call it,
including return_row_major, which milestone 3 needs because the norm+gate kernel
consumes ROW_MAJOR directly and the default TILE relayout would be work done
only to be undone.

And it applied, imported and re-ran idempotently inside vllm-tt:k1.

Two behaviours carried deliberately from the unpacked path rather than
reinvented: a one-shot engagement print, so a control arm that silently engaged
is detectable; and a resolve-once-then-warn-and-return-None cache, so an absent
binding falls back instead of raising.

The op is not wired into tp.py yet -- that is wire-k2's surgery, next.
WHERE-TO-OPTIMISE says where the time goes; this says what we do about it,
layer by layer. Layers 1, 2, 4 get measurements (CCL share, DRAM saturation
curve, op-count probe + post-K profile) because their headroom is either
unknown or not ours to move; layer 3 gets the programme (K2-K5, M, state->L1,
ops/GDN layer 41 -> <=22); layer 5 gets the batch curve; layer 6 gets nothing
for speed and stays a GSM8K gate. Execution order is by expected ms per hour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The exec_sync driver ran the pytest demo bare and would have died looking
for Qwen/Qwen3.6-27B online: nothing but vLLM sets HF_MODEL, the mesh and the
caches. It now carries the goal3 profile's environment (bf4 MLP down
included) so device time is measured on the stack every record was set on.

K2 overlays the ttbuild .so pair that has patch_packed applied, ships the
changed reader kernel source, installs our 0012 wrapper, and copies tp.py
wired by the external wire-k2 payload. Staging trap recorded in the
Dockerfile: build_Release/lib/*.so is stale in the warm tree, ninja writes
build_Release/ttnn/*.so -- the lib copy has no packed symbol and would have
shipped as a null-result arm. Assertions check the binding, its signature,
and that the reader on disk is the packed one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 102.06 aggregate was measured with 3c deliberately off; the all-four
stack's decode number does not exist yet. Say so where the number lives.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Decides whether the 21.7 ms floor is reachable from the 33-core grids decode
matmuls run on, and whether lever H was closed for the right reason. Runs
when the device is free; the harness owns it right now.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The device-time instrument resolves in one pass what three endpoint passes
could not. Its side effect is the larger result: the demo does a token in
~49.8 ms where vLLM does the same token in ~62, so a fifth of single-stream
latency is serving overhead. Recorded as a layer-5 item.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The runner measures; the ledger judges. One arm per candidate against the
current best, engagement asserted per flag, adopt at >=1%, park under 0.5%,
discard anything worse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
patch_fuse_ng anchors on a cbd namespace that only patch_state_fast (not
adopted) introduces; the copy under patches/k has those anchors adapted to
our cb and applies fully to the K2 tree. 0013 gives our wrapper the four
new kwargs and skips the relayout when the op returns the gated tensor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Operator pivot 2026-09-05. Carries the 27B programme's instruments and loop,
drops its GDN levers, states inferred targets (60 / 300 tok/s, 8k prefill)
so they can be measured wrong, and lists the MoE lever candidates with
provenance. Phase 0 (scouting upstream presets) is running in parallel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bundle: Adartras/qwen3.6-a3b-blackhole (HF, 42a7103), single p150, ttnn>=0.77.
Ornith-1.5-35B-A3B is architecturally identical to Qwen3.6-35B-A3B (config diff:
pad_token_id, router_aux_loss_coef, use_cache only). Three deviations, each
its own file: a ttl import shim (tt-lang is cp312-only and pins ttnn 0.74),
an FP8 compressed-tensors loader patch that dequants per-channel and
assembles per-expert tensors into the fused [E,2I,H]/[E,H,I] the bundle
expects, and a hand-written vllm_metadata.json for the plugin's
EXTRA_MODELS_DIR hook.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ornith == Qwen3.6-35B-A3B (three trivial config keys differ). Active params
2.94B, of which only a third is experts; 2.75 GB/token at the recipe's
dtypes -> 6.4 ms floor on one card, and the recipe's author sits at 4.5x
that floor -- the 27B's op-floor signature again. Existing-presets table:
the Adartras tt-kernel bundle is the only Qwen3.5-MoE implementation found
anywhere; tt-metal upstream has none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
goal 7: tt-forge-models Qwen3.6 loader is a verified 2-card sharding blueprint, not a serving path; smoke script
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 4s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 5s
Details
fe1c0782b0 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
goal 7: FP8 loader's gate-first fusion and dequant verified against the bf16 checkpoint (cos 0.9996)
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 4s
Details
746e17f8c8 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
goal 7: first device command is the bundle's own demo (model-level ms/token), vLLM after
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 4s
Details
169b725ecd 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Grok commented 2026-09-25 10:06:28 +02:00
Owner
Copy link

Closing: the branch agent/claude-auth/ornith-moe is fully merged into master (ancestor), and the project chose Qwen3.8-27B / ThinkingCap over this goal. It is being archived as tag archive/agent/claude-auth/ornith-moe.

Ornith-1.5-35B-A3B remains a candidate for the new 1-card target, tracked on #69. Its 32.5 tok/s single-P150 figure is third-party and was never measured here.

Closing: the branch `agent/claude-auth/ornith-moe` is fully merged into master (ancestor), and the project chose Qwen3.8-27B / ThinkingCap over this goal. It is being archived as tag `archive/agent/claude-auth/ornith-moe`. Ornith-1.5-35B-A3B remains a candidate for the new 1-card target, tracked on #69. Its 32.5 tok/s single-P150 figure is third-party and was never measured here.
Grok closed this pull request 2026-09-25 10:06:28 +02:00
Grok commented 2026-09-25 10:06:30 +02:00
Owner
Copy link

agent-gateway closeout (closed): cleared agent:* labels.
Hermes kanban card (if any) is left for Hermes to archive/complete.

agent-gateway closeout (closed): cleared `agent:*` labels. Hermes kanban card (if any) is left for Hermes to archive/complete.
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 4s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack!16
No description provided.