bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

WIP: goals 2 and 3 - long context, and past the external reference #14

Closed
multica-agent wants to merge 19 commits from agent/claude-auth/goal-2-3 into agent/fable/cxx-fork-build
pull from: agent/claude-auth/goal-2-3
merge into: bitpartner:agent/fable/cxx-fork-build
Conversation 1 Commits 19 Files changed 13 +968 -54
multica-agent commented 2026-09-05 00:02:45 +02:00
Member
Copy link

Stacked on #13 (agent/fable/cxx-fork-build), which is where the C++ prefill
fixes live. Merge that first.

Goal 1 is closed out

14.368 tok/s, and 14.418 tok/s on a second run after a full container
teardown and a warm restart — 0.3% apart, which settles goal 1's third
condition (reproducible without hand-holding) by measurement rather than
assertion. Both in bench/runs/goal1-endpoint-fp8-2026-09-04.jsonl.

decode-bench.py no longer hardcodes the retired 18 tok/s target; the
threshold is --target, default 10. It also referred to args.target where
the namespace is called a, so my own threshold change crashed it on first
use — fixed, and that fix is why there is a second baseline number at all.

Goals 2 and 3 now have gates

They existed only as sentences in goal 1's "explicitly not in goal 1" list.
Two findings from writing them down properly:

  • Goal 2 is bounded by bf8 KV, not by cleverness. 4 × 256k is 1.05 M
    tokens of KV; at bfloat16 that is 64 GiB against 64 GiB of total device
    DRAM, and at bfloat8_b it is 32.5 GiB. So QWEN_SDPA_BF8 is load-bearing,
    and the ~1.4 GiB of margin it leaves is thin enough that goal 3's work
    belongs first     — down_proj at bfloat4_b returns 2.65 GiB of DRAM. The
    ordering runs opposite to the numbering.
  • Goal 3 cannot be one change. Decode is already traced
    (qwen36_vllm.py captures at pos 0 and replays per bucket), and goal 1's
    p99/p50 of 1.18 says there is no stall left to remove. So the dispatch lever
    is spent and only the divisor is left — and down_proj alone projects to
    ~15.9 tok/s against a >16.0 gate.

Patch 0008

bfloat4_b on mlp.down_proj, gated on QWEN36_MLP_DOWN_BF4: 5.704 G params,
1.0625 → 0.5625 bytes/element, −2.65 GiB per token, ceiling 29.1 → 32.2 tok/s.
Chosen because two independent published NVFP4 recipes for this model quantise
down_proj — evidence-led rather than the first candidate to hand. Verified
on device: the logs now read mlp.down_proj.weight.tp_dtype_BFLOAT4_B.

Four things went wrong, and each is recorded where it will be re-read

  1. The gate was inert where it mattered. load_mlp_weights() returns
    early at num_devices > 1 through tp_common.shard_w. The first version
    patched only the single-device branch — the one that reads naturally when
    you skim — and the image built, imported, asserted its own gate, loaded 64
    layers and served, while every log line said BFLOAT8_B. A null result
    that looks like a finding is the worst outcome available, so the overlay
    now counts four occurrences of _down_dtype and fails the build
    otherwise.
  2. I invented a trap that does not exist. The claim was that the MLP cache
    file carries no dtype, so the flip would silently reload the bf8 tensors.
    False — ttnn.as_tensor appends dtype and layout to the filename, so the
    two dtypes are separate files that coexist and switching is warm in both
    directions. Corrected in place rather than quietly dropped. The design is
    better than I assumed, which makes the dtype sweep cheap: each candidate
    materialises once, ever.
  3. The overlay globbed the patch directory and re-applied 0001–0004, which
    are already merged into the fork branch. They matched cleanly in neither
    direction, patch(1) took them with fuzz, and the image died at
    NameError: name 'Mode' is not defined. patch -R --dry-run is not a
    reliable already-applied test on a rebased tree. Now: named files,
    --fuzz=0, and a real import before the image ships.
  4. A new dtype needs one cold build. Patch 0005's guard caught the warm
    path about to tilize NaN placeholders into the new bf4 filenames, deleted
    them and refused to start. That guard saved the experiment; COLD=1 is now
    a documented switch.

Also

  • scripts/vllm-tt-serve.sh — the working invocation, in the repo at last,
    with goal1 / goal2 / goal3 profiles. It lived only in a shell history
    and a container's Config.Cmd.
  • bench/long-context-bench.py — N concurrent streams at a real depth, since
    256k behaviour cannot be inferred from 4k on an architecture where 16 of 64
    layers scale with context and 48 do not.
  • docs/FORKS.md — separates the C++ patches (overnight wheel build) from the
    Python ones (seconds, last image layer).

Still open

The goal-3 number itself: the cold bf4 build is running on cfx-llm2 now. WIP
until that measurement is in bench/runs/, and goal 2's ladder has not
started.

Tier: T1 (a dtype policy change behind an env gate, plus docs and scripts).

Stacked on #13 (`agent/fable/cxx-fork-build`), which is where the C++ prefill fixes live. Merge that first. ## Goal 1 is closed out **14.368 tok/s**, and **14.418 tok/s** on a second run after a full container teardown and a warm restart — 0.3% apart, which settles goal 1's third condition (reproducible without hand-holding) by measurement rather than assertion. Both in `bench/runs/goal1-endpoint-fp8-2026-09-04.jsonl`. `decode-bench.py` no longer hardcodes the retired 18 tok/s target; the threshold is `--target`, default 10. It also referred to `args.target` where the namespace is called `a`, so my own threshold change crashed it on first use — fixed, and that fix is why there is a second baseline number at all. ## Goals 2 and 3 now have gates They existed only as sentences in goal 1's "explicitly not in goal 1" list. Two findings from writing them down properly: - **Goal 2 is bounded by bf8 KV, not by cleverness.** 4 × 256k is 1.05 M tokens of KV; at `bfloat16` that is 64 GiB against 64 GiB of total device DRAM, and at `bfloat8_b` it is 32.5 GiB. So `QWEN_SDPA_BF8` is load-bearing, and the ~1.4 GiB of margin it leaves is thin enough that **goal 3's work belongs first** — `down_proj` at `bfloat4_b` returns 2.65 GiB of DRAM. The ordering runs opposite to the numbering. - **Goal 3 cannot be one change.** Decode is *already traced* (`qwen36_vllm.py` captures at pos 0 and replays per bucket), and goal 1's p99/p50 of 1.18 says there is no stall left to remove. So the dispatch lever is spent and only the divisor is left — and `down_proj` alone projects to ~15.9 tok/s against a >16.0 gate. ## Patch 0008 `bfloat4_b` on `mlp.down_proj`, gated on `QWEN36_MLP_DOWN_BF4`: 5.704 G params, 1.0625 → 0.5625 bytes/element, −2.65 GiB per token, ceiling 29.1 → 32.2 tok/s. Chosen because two independent published NVFP4 recipes for this model quantise `down_proj` — evidence-led rather than the first candidate to hand. Verified on device: the logs now read `mlp.down_proj.weight.tp_dtype_BFLOAT4_B`. ## Four things went wrong, and each is recorded where it will be re-read 1. **The gate was inert where it mattered.** `load_mlp_weights()` returns early at `num_devices > 1` through `tp_common.shard_w`. The first version patched only the single-device branch — the one that reads naturally when you skim — and the image built, imported, asserted its own gate, loaded 64 layers and served, while every log line said `BFLOAT8_B`. A null result that looks like a finding is the worst outcome available, so the overlay now counts **four** occurrences of `_down_dtype` and fails the build otherwise. 2. **I invented a trap that does not exist.** The claim was that the MLP cache file carries no dtype, so the flip would silently reload the bf8 tensors. False — `ttnn.as_tensor` appends dtype and layout to the filename, so the two dtypes are separate files that coexist and switching is warm in both directions. Corrected in place rather than quietly dropped. The design is better than I assumed, which makes the dtype sweep cheap: each candidate materialises once, ever. 3. **The overlay globbed the patch directory** and re-applied 0001–0004, which are already merged into the fork branch. They matched cleanly in neither direction, `patch(1)` took them with fuzz, and the image died at `NameError: name 'Mode' is not defined`. `patch -R --dry-run` is not a reliable already-applied test on a rebased tree. Now: named files, `--fuzz=0`, and a real import before the image ships. 4. **A new dtype needs one cold build.** Patch 0005's guard caught the warm path about to tilize NaN placeholders into the new bf4 filenames, deleted them and refused to start. That guard saved the experiment; `COLD=1` is now a documented switch. ## Also - `scripts/vllm-tt-serve.sh` — the working invocation, in the repo at last, with `goal1` / `goal2` / `goal3` profiles. It lived only in a shell history and a container's `Config.Cmd`. - `bench/long-context-bench.py` — N concurrent streams at a real depth, since 256k behaviour cannot be inferred from 4k on an architecture where 16 of 64 layers scale with context and 48 do not. - `docs/FORKS.md` — separates the C++ patches (overnight wheel build) from the Python ones (seconds, last image layer). ## Still open The goal-3 number itself: the cold bf4 build is running on cfx-llm2 now. WIP until that measurement is in `bench/runs/`, and goal 2's ladder has not started. Tier: T1 (a dtype policy change behind an env gate, plus docs and scripts).
The endpoint served 256 streamed tokens at 14.368 tok/s single-stream decode
on Qwen3.8-27B-FP8 across both p150a at TP=2. Goal 1 asks for >= 10, so this
passes; decode-bench.py printed FAIL only because it still hardcoded the 18
that the operator retired earlier the same day. The threshold is now --target
with a default of 10, so the script measures and the goal document decides.

Recorded with the distribution, not just the mean, because the shape is the
interesting part: p50 69.8 ms, p99 82.0 ms, max 88.4 ms. A p99/p50 of 1.18
means no periodic stall -- the ~51% of DRAM bandwidth we are leaving unused is
systematic per-token overhead, which is what goals 3's levers have to attack.
Goal 1's 14.368 tok/s is ~49% of the measured DRAM bandwidth, and decode is
already traced -- so the remaining overhead is not dispatch and the cheap MBU
levers are spent. The way past the external reference's 16.0 tok/s is to make
the divisor smaller, not the utilisation higher.

down_proj is the obvious first cut: 5.704 G params, the last tensor of MLP
size still at bfloat8_b now that gate/up are bfloat4_b. At bfloat4_b it drops
2.65 GiB from the 27.49 GiB per-token read, moving the 100%-MBU ceiling from
29.1 to 32.2 tok/s and freeing 2.65 GiB of device DRAM that goal 2's 4x256k KV
budget wants. Two published NVFP4 recipes for this model quantise down_proj,
so this is the evidence-led candidate rather than the first one to hand.

Patch 0008 gates it on QWEN36_MLP_DOWN_BF4 -- upstream's 'down: bfloat8_b
(accuracy)' is a deliberate choice and it has to survive a greedy comparison
before it becomes our default. The second hunk is the one that would have
wasted the whole experiment: the MLP cache file name carries no dtype, so
without adding the gate to build_variant() the warm cache would have handed
back the old bfloat8_b tensors while reporting a complete build, and the
benchmark would have shown no change for the wrong reason.

The overlay Dockerfile exists because the wheel stage is an overnight compile
here and our build cache was pruned; it applies Python patches onto an existing
image, and refuses to produce one where the patch did not land.

scripts/vllm-tt-serve.sh finally puts the working invocation in the repo with
the three profiles, which is goal 1's third condition.
The first overlay build succeeded and produced a broken image. It globbed
patches/tt-metal/*.patch, but 0001-0004 are already merged into the fork
branch the base image is built from, so it re-applied them onto a tree that
already had them. They matched cleanly in neither direction, patch(1) took
them with fuzz regardless, and the result lost the `Mode` import that 0003
touches. The container then died at import with `NameError: name 'Mode' is
not defined` -- nowhere near the dtype change under test, and only after a
device open and several minutes of engine init.

Three changes, each aimed at that failure rather than at its symptom:

- the COPY names the one patch that is not in the base, so 'which patches does
  this image add' is answered by reading the Dockerfile;
- --fuzz=0 --forward, so a patch that no longer matches its context fails the
  build instead of being guessed at;
- a real import of qwen36_vllm plus an assertion that the gate reaches
  build_variant(). The grep proves text landed; only an import proves the tree
  still parses, and only the assertion proves the cache will be invalidated.

0008 is regenerated with diff -u against the pristine tree inside vllm-tt:src
instead of being hand-written, so it now applies at zero fuzz. Verified in the
image: import clean, gate True.

The transferable part: `patch -R --dry-run` is not a reliable already-applied
test on a tree the patch was rebased across.
Both existed only as sentences inside goal 1's 'explicitly not in goal 1'
list, which was fine while goal 1 was unmet and is not fine now that it is.
Each now has a gate that passes or does not, the arithmetic that says whether
it is reachable, and an ordered list of what is likely to stop us.

Two findings from writing them down are worth more than the prose:

Goal 2 is bounded by bf8 KV, not by cleverness. 4 x 256k is 1.05 M tokens of
KV; at bfloat16 that is 64 GiB against 64 GiB of total device DRAM, and at
bfloat8_b it is 32.5 GiB. QWEN_SDPA_BF8 is therefore load-bearing rather than
an optimisation -- and the ~1.4 GiB of margin it leaves is thin enough that
goal 3's down_proj change, which returns 2.65 GiB of DRAM, belongs first. That
is the ordering argument between the two goals, and it runs the opposite way
to their numbering.

Goal 3 cannot be one change. Decode is ALREADY traced -- qwen36_vllm.py
captures at pos 0 and replays per bucket -- so the dispatch lever a naive
implementation would start with is spent, and goal 1's p99/p50 of 1.18 says
there is no stall left to remove. What remains is the divisor, and down_proj
at bfloat4_b lands at ~15.9 tok/s against a > 16.0 gate. Stating that up front
beats discovering it after the run.

Also records goal 2 as a pooled 1,048,576-token budget rather than a 4x256k
product, because QWEN36_MAX_TOKENS_ALL_USERS is how the model itself expresses
concurrency x context, and the pool is simultaneously 4x256k, 8x128k and
16x64k.
Three corrections, all found by running the thing rather than reading it.

The gate was inert where it mattered. 0008 changed only load()'s single-device
path; on a two-card mesh load_mlp_weights returns early through tpc.shard_w,
which has its own two down_proj call sites. The image built, imported,
asserted its own gate, loaded 64 layers and served -- while every log line
read mlp.down_proj.weight.tp_dtype_BFLOAT8_B. _down_dtype is now hoisted to
the top of the function and the overlay counts four occurrences, so one
patched site out of three fails the build instead of producing a null result
that looks like a finding.

The tensor cache does not collide across dtypes, contrary to what the first
version of this patch and the goal 3 section both claimed. ttnn's as_tensor
appends the dtype to the filename it writes, so bf8 and bf4 are separate
files and the flag is warm in both directions once each set exists. The
build_variant hunk stays -- the marker certifies a variant's file set is
complete, and bf4 is a different set -- but the reasoning behind it was wrong
and is corrected in place rather than quietly kept.

decode-bench.py referred to args.target where the namespace is called a, so
the threshold change I made two commits ago crashed the script on first use.
Fixed, and the fix is why there is a second baseline number at all.

That second number is the useful one: 14.418 tok/s against 14.368 earlier,
0.3% apart, after a full container teardown and a warm start from the tensor
cache. Goal 1's third condition -- reproducible, comes back without
hand-holding -- is now measured rather than assumed.
long-context-bench.py answers the two questions decode-bench.py cannot: does
the KV allocation hold at depth, and what does depth cost. You cannot infer
either from 4k on this architecture -- 16 of 64 layers scale with context and
48 do not -- so it builds a real prompt of a real length and runs N of them
concurrently. TTFT with one stream and no queue IS the prefill measurement,
which is the number goal 2 is most likely to be judged on, since prefill at
256k on four Zen 1 cores is expected to be minutes.

It ladders on purpose. Each distinct prefill width compiles on first use, so
32k -> 128k -> 256k costs more wall clock than 256k alone, and buys a failure
that is cheap to read instead of one that costs an hour to reproduce.

The doc correction matters more than the script. Goal 3's section claimed the
MLP tensor cache would silently hand back bfloat8_b tensors under a bfloat4_b
flag because the cache file name carries no dtype. That was wrong -- ttnn's
as_tensor appends dtype and layout to the name it writes, so the two dtypes are
separate files that coexist, and switching the flag is a warm start in both
directions. The design is better than I assumed, and the dtype sweep is
correspondingly cheap: each candidate materialises once, ever.

Replaced with the trap that was real: at TP=2 load_mlp_weights returns early
through tp_common.shard_w, so the branch that reads naturally when you skim
the function is not the branch that runs. That one produced a serving model
whose logs said BFLOAT8_B under a flag asserted at build time.
The bf4 down_proj run loaded all 64 layers and then refused to start:

  RuntimeError: Warm build generated 64 new tensorbins from placeholders
  (e.g. layers.0/mlp.down_proj.weight.tp_dtype_BFLOAT4_B_layout_TILE...);
  they were deleted. The marker did not cover this build.

That is patch 0005's guard doing exactly what it was written for, and it is
worth being clear that it saved the experiment rather than blocking it. The
warm path substitutes dataless NaN placeholders for device weights, because on
a warm start the real values are already tilized on disk. A dtype change
creates filenames that do not exist yet, so the warm path would have happily
tilized the placeholders and served a model of NaNs -- which at best crashes
and at worst produces plausible-looking garbage.

COLD=1 sets TT_TRANSFORMERS_FORCE_MODEL_LOAD=1, needed exactly once per new
tensor-cache variant. After that the bf4 set is on disk and, because ttnn keys
cache files on dtype, both dtypes stay warm and switching between them is
free. The comment in the script says all of this, since the error message is
the kind that reads like a defect at 2am.
The two kinds were undifferentiated, which made every patch look like it
needed the overnight wheel compile. 0005-0008 are Python and land in the
image's last layer in seconds; only 0004 and the two upstream PRs touch the
wheel. Recording which is which is what makes the overlay Dockerfile a
legitimate tool rather than a shortcut, and it comes with the constraint that
keeps it honest: both Dockerfiles must list the same Python patches.

Also records the three failure modes from today, because each cost real time
and none is guessable: globbing the patch directory onto a fork-built tree
re-applies the patches already merged into the fork branch; hand-written hunk
headers apply with fuzz and then fail under --fuzz=0; and a dtype change needs
exactly one COLD=1 build or the warm path tilizes NaN placeholders into the
new filenames.
Every MBU figure in this repo took 27.49 GiB from the external report. Summing
our own tensor-cache files by name group says that is wrong, because two of
the largest cached tensors are not read per token:

  mlp.gate_up.weight.swiglu.tp   5.98 GiB  packed [gate|up] for the fused
                                           PREFILL agmm -- the source says
                                           decode keeps w1/w3
  tok_embeddings.weight          2.37 GiB  a row lookup, not a matmul

The corrected read is 20.17 GiB, and it moves both numbers in opposite
directions: the 100%-MBU ceiling rises from 29.1 to 39.7 tok/s, and our
measured 14.42 tok/s is 36% of DRAM rather than 49%.

That is better news and worse news, and the worse half is the interesting one.
More headroom exists than documented. But a decode that is ALREADY TRACED,
with a p99/p50 of 1.18 and no episodic tail, sitting at 36% MBU, has something
systematic in front of every token -- which is not the conclusion the previous
revision drew when it said the dispatch levers were spent. Three candidates
are now written down with the evidence for each: host round-trips (_ondev_argmax
is False, so logits cross PCIe Gen3 x4 every token), M=1 matmul efficiency on a
path tuned for 1 and 4 cards, and the per-layer CCL at a TP=2 that falls back
to TP=4 constants.

It also changes the goal 3 projection from a miss to a pass: at a constant 36%
MBU, down_proj at bfloat4_b lands at 16.6 tok/s against the > 16.0 gate rather
than the ~15.9 the old divisor predicted. Whether MBU holds constant is the
open question, which is the argument for measuring instead of projecting.
14.864 tok/s against a 14.418 baseline. The read shrank 13.2% and the time
shrank 3.0%, so MBU FELL, from 36.0% to 32.5%. That is the opposite of what a
bandwidth-bound decode does when you hand it fewer bytes, and it is the most
useful number produced today.

Fit t = fixed + read/B across the two points: ~54 ms of the 67 ms per token is
NOT streaming weights. Roughly 80% fixed cost. The consequence is arithmetic
rather than opinion -- moving every remaining bfloat8_b group to bfloat4_b
(qkvzab 4.02, GDN out 1.49, lm_head 1.26, wqkv 1.16, wo 0.50 GiB) buys 3.97
GiB, about 3.1 ms on the measured slope, about 15.6 tok/s. Still short of 16.0.
So no reachable dtype policy closes goal 3's gap, and continuing down that
path would have burned a cold build per candidate to find that out one
candidate at a time.

Where the fixed cost is NOT: I had written host logits readback as the leading
suspect on the grounds that model.py sets _ondev_argmax = False. That is the
wrong read of the code -- the vLLM decode path calls _forward_decode with
sharded_lm_head=True, which takes the same pre-gather branch, so the
all-gather and full-logits readback are already skipped. Removing a suspect
counts.

Also adds dtype-accuracy-check.py, which scores single next-token predictions
across 24 strongly-determined prompts. The obvious design -- echo the prompt
and diff per-token logprobs -- is unavailable: the TT plugin answers
'Not yet supporting prompt_logprobs on tt'. And diffing two greedy generations
is worse than useless: they diverge at the first perturbed token and every
token after differs by construction, which scored this change at 6% agreement
when nothing had been shown to be wrong with it.
Replaces the projection with the measurement. down_proj at bfloat4_b cut the
per-token read 13.2% and bought 3.1%, and MBU fell from 36.0% to 32.5% -- a
bandwidth-bound decode handed 13% fewer bytes does not behave that way.
Fitting the two points puts ~54 ms of the 67 ms per token outside the weight
stream, and halving EVERY remaining bfloat8_b group buys ~3.1 ms on that
slope, landing at ~15.6 tok/s against a 16.0 gate -- while optimistically
assuming lm_head and the attention projections tolerate 4 bits, which they
probably do not.

Better to write that down now than to spend one cold build per candidate
discovering it one candidate at a time.

Records where the cost is not, since a ruled-out suspect is worth keeping:
_ondev_argmax reads False, but the vLLM path calls _forward_decode with
sharded_lm_head=True and takes the same pre-gather branch, so logits readback
was never happening. Decode is traced with no episodic tail, so it is not
per-op dispatch either.

The hypothesis that survives is architectural: 48 of 64 layers are Gated
DeltaNet, their per-token work is a fixed-size recurrent state update, and
fixed-size means latency-bound -- small tensors in a serial chain, nothing to
amortise. ~1.1 ms per GDN layer is the right order. If that holds, the
property that makes 256k context affordable is the same one that makes
short-context decode slow, which is a genuinely interesting trade and not one
either goal anticipated. Flagged as needing a profile before more code.

down_proj bf4 stays on for goal 2 regardless, on memory grounds: 2.65 GiB of
DRAM is the difference between ~0.3 and ~3 GiB of KV margin at 4x256k.
32k x 1: prefill 9.18 s, decode 14.693 tok/s, 513 KV blocks allocated.

Both numbers correct something. The goal 2 section listed prefill time as the
most likely blocker, on the strength of goal 1's 6m06s prefill warmup at chunk
2048. That was a bad generalisation: the warmup is compilation, and steady
prefill runs at ~3270 tok/s, which puts 256k at roughly 80 s. The risk list had
its first item wrong.

And decode at 32k is 1.1% below decode at 4k -- 14.693 against 14.864 -- for
eight times the context. That is the Gated DeltaNet structure behaving exactly
as VRAM-BUDGET.md said it would, with only 16 of 64 layers carrying a
context-scaling cache. It also sits oddly next to goal 3's finding: the same
architecture that makes context nearly free is the one leaving decode at 32%
MBU.
Prefill time was listed first on the strength of goal 1's 6m06s warmup at
chunk 2048. The 32k rung measured 9.18 s for the same work at eight times the
length, because the warmup is compilation and not throughput -- so prefill
drops from first to last, and 256k extrapolates to ~80 s.

The allocation at the full 1.05 M-token pool moves to first, being the only
item that can fail outright rather than be slow, and bf8 KV accuracy at depth
moves to second because nothing measured so far tests it -- the 32k rung was
throughput, not quality. Concurrency-4 aggregate throughput is genuinely
unknown and now says so instead of being assumed fine: at ~80% fixed cost per
token, four streams might batch well or might contend.
Both remaining suspects for goal 3's ~54 ms fixed cost -- the 48 sequential
Gated DeltaNet state updates, and the per-layer all-reduce at TP=2 -- fit the
evidence equally well, and a device profile is the thorough way to separate
them. TP=1 on a single card is the cheap way: it removes every CCL round while
leaving the GDN chain untouched, and the weights fit, since down_proj at
bfloat4_b puts on-device resident weights at ~25.9 GiB against one card's 32.

Unchanged decode means the all-reduce is innocent and the work is in the GDN
kernel; substantially faster means TP=2's CCL is the cost and the TP=4 tuning
fallback recorded in TP2-AND-TT-FORGE.md becomes the target. Either answer
constrains what a profile has to explain, for one cold build.

Written down with the caveat that TP=1 is a diagnostic and not a candidate: it
halves available bandwidth and cannot hold goal 2's KV pool, so a good number
there is a finding about overhead, not a proposal to serve that way.
From the allocator's own numbers when 4 x 256k died in warmup: 8 banks of
4,142,923,648 B is 30.87 GiB per card, 61.73 GiB across the pair. The
spec-sheet 32 GiB is not all addressable, and the shortfall is bigger than the
margin this document's plans were built on.

It is exactly why 4 x 256k failed. The KV pool of 34.00 GiB allocated
successfully against 33.83 GiB of real headroom, left 179 MB, and warmup then
asked for 80 MiB it could not get -- with a largest free block of 10.2 MB, so
fragmentation as well as exhaustion.

Also records how to read that message, since it is the diagnostic for every
future device OOM on this host: bank size x 8 is capacity, allocated is what
is spoken for, and largest free block separates real exhaustion from
fragmentation.
Four concurrent 262,144-token requests allocated, served and came back
coherent. vLLM's own accounting: GPU KV cache size 1,048,832 tokens, maximum
concurrency 4.00x. That is goal 2's gate, functionally.

It needed BOTH memory levers together -- down_proj at bfloat4_b for 2.65 GiB
and the trace region halved for 1 GiB -- because usable DRAM is 30.87 GiB per
card rather than 32. With either one missing the warmup OOMs.

The decode numbers from that run are junk and the harness is why: prefill
SERIALISES. Stream 1 starts decoding while streams 2-4 are still prefilling,
so its inter-token gaps are other requests' prefill chunks, which reported
1.562 tok/s per stream against a true single-stream 14.7 and made the
aggregate look like a regression. With --gen 32 against ~2-minute prefills the
steady-state phase is a rounding error in the window. Documented in the
script: above concurrency 1, trust 'did it allocate', 'did it stay up' and
'is the text coherent', and use --gen 256+ for a rate.

The coherence is not an artefact, and it is the first evidence on the question
upstream flags with 'validate PCC at long ctx': the model read a ~240k-token
prompt and correctly described its structure with bf8 KV.
At 4 x 256k the four TTFTs came back 135.8, 270.8, 405.8 and 540.6 s -- exactly
135 s apart, because prefill serialises: one request at a time, ~135 s per
240k prompt. So each stream's own decode rate is mostly a record of how long it
sat interleaved with other requests' prefill chunks, and the four rates read
0.366, 0.537, 1.007 and 8.128 tok/s for four streams doing identical work.
Averaging those describes the queue, not the engine.

The bench now keeps absolute per-token timestamps and reports a steady-state
aggregate over [max(first token), min(last token)] -- the window in which every
stream was actually decoding. It prints that as the throughput number and says
so, with the per-stream averages demoted.

The last stream is the reason this is worth measuring rather than estimating.
It decoded at 8.128 tok/s with all four active, which implies ~32 tok/s
aggregate against 14.86 single-stream. If that holds it is a 2.2x batching win,
and it is exactly what goal 3's finding predicts: decode is ~80% fixed cost per
token, and a fixed cost is precisely what batching amortises. The corollary is
that the 36% MBU single-stream figure is not the machine's ceiling -- it is the
cost of running one stream at a time.
Measured over the 19.1 s window in which all four streams were decoding: 617
tokens, 32.242 tok/s aggregate, 8.06 per stream. Against 14.864 single-stream
that is a 2.17x batching win, and it moves MBU from 32.5% to 70.5% of the
measured 860 GB/s.

This is the same fact goal 3 found, read forwards. Goal 3 measured ~54 ms of
the 67 ms per token sitting outside the weight stream and concluded no dtype
policy could close an 11% single-stream gap. A per-token fixed cost is
precisely what batching amortises -- four streams share one weight read per
step -- so the two results are one result: 32.5% MBU is not the hardware's
ceiling, it is the cost of running one stream at a time.

Worth stating plainly because it changes what to optimise. The single-stream
number is the one that misses its gate; the number that matters for a serving
host is the aggregate, and that one is at 70.5% of DRAM.

Recorded with what the naive averages said -- 2.502 per stream, 10.006
aggregate -- because they look like a 4x concurrency regression and are purely
an artefact of prefill serialising at ~135 s per 240k prompt. The four TTFTs
came back 135 s apart to within a second.
The trace region is pure memory -- it bounds how many decode traces can be
held, not how fast they replay. At 1024 MiB per device the two regions are
2 GiB of the 61.73 GiB total, and the first 4x256k attempt died 0.17 GiB short
of its 34.00 GiB KV pool. Halving it is what made the run fit.

Leaving the default at 1024 and the passing configuration undescribed would
have made goal 2 unreproducible from this script, which is the thing the
script exists to prevent.
multica-agent commented 2026-09-05 19:05:57 +02:00
Author
Member
Copy link

Closing as obsolete: this was a draft/WIP PR for goals 2-3 that has been superseded by PR #15 (agent/claude-auth/goal-4-nvfp4-single-card), which includes all of these commits and continues the work.

Closing as obsolete: this was a draft/WIP PR for goals 2-3 that has been superseded by PR #15 (`agent/claude-auth/goal-4-nvfp4-single-card`), which includes all of these commits and continues the work.
multica-agent closed this pull request 2026-09-05 19:06:01 +02:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack!14
No description provided.