bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

New issue
Open
opened 2026-09-23 21:34:26 +02:00 by Grok · 4 comments
Grok commented 2026-09-23 21:34:26 +02:00
Owner
Copy link

Status 2026-09-23. Single-stream Qwen3.8-27B, TP=2, two P150a, no MTP. Gate is ≤25.0 ms/token on the 4k/256 protocol. Not met.

Serving stays stopped for this whole campaign (tenstorrent-serving.service inactive). Do not start it. Do not docker rm -f a live tt-metal container. One tt-metal process. Lock /tmp/ttlock.

Measured this campaign

  • Control bench/runs/p150-decode-40-control-20260923T1652Z.jsonl: 4k/256 48.619 ms (20.57 tok/s), 16k 49.707 ms, text sha256 bafed7a9f10786fe726a5f7d3bea77cfd18fe22262edeb0b9c5265ebd9b0b2ea. Same boot, max_model_len=34816, max_num_seqs=1, prefix caching off, speculative config null. Arm env: QWEN36_DECODE_LOGITS_RM=1, QWEN36_LM_HEAD_GATHER_MODE=none, QWEN36_1D_GRID_ATTN_QKV=11x4.
  • Hypothesis A closed, bench/runs/p150-decode-40-hypA-hostdev-20260923T1705Z.jsonl: 48.563 ms, same sha. py-spy, 1799 samples, step 48.563 ms: ttnn.decorators.__call__ leaf 45.297 ms (93.3%), other host leaves 3.266 ms, process_output_decode 0.486 ms. The itemiser text that says "~47 ms host" is wrong. That frame is a blocked native call. The device-wait hint list does not name it.

Why ≤25.0 ms is the edge

N1 bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl: the seven production decode projections, standalone, trace, M=32. 30.381 ms = 22.132 ms DRAM + 8.249 ms dtype-independent floor. Arithmetic is <2 ms (LoFi→HiFi4 barely moves). Marginal read rate 454–469 GB/s on the 11-wide grids. Fit error about ±1.5 ms. These are standalone ceilings, not in-model savings.

45.3 ms is inside one traced ttnn call. Subtracting the 30.4 ms standalone matmuls leaves ~15 ms of other device work in that call (in-model contention can move this by a few ms; do not treat 15 as exact). Host is 3.3 ms.

  • Zero the 8.2 ms floor only → about 40 ms. Short of the gate.
  • Zero the floor and the ~15 ms, and pay the host → about 25 ms, i.e. the DRAM read of those seven projections plus the host tail. lm_head bytes are not in the 22.1 ms. So 25.0 ms requires the floor and the non-matmul device work to hide inside the weight read, not merely to get smaller one at a time.

Closed — do not redo

  • A, host vs device. Host is 3.3 ms.
  • Trace-off. Replay is already on; turning it off costs ~176 ms/step at B=8.
  • Op-count reductions. Dispatch is already amortised inside the trace.
  • QWEN36_1D_GRID_* tuning. Inside a 0.5% band. ATTN_QKV=11x4 is already the arm.
  • More bf4. gate/up are already bf4. down_proj bf8 is deliberate.
  • LM_HEAD_GATHER_MODE=none as a new win. It is already in the 48.6 ms number. It regresses by ~31 ms at B=8. Do not "turn it on".
  • The P0 / docs/WHERE-THE-BOTTLENECK-IS.md op table (4,312 programs, layout glue, CCL share). That profile ran the composed unfused GDN path, not this binary. Do not cite those shares as production.

Open children

Filed separately. Each one names a falsifier. A boot happens only after that falsifier is a single change or a single measurement, and only one boot at a time.

Any new serving number must interleave against this control: control lands 47–52 ms, sha matches bafed7a9…b2ea, both banners present (QWEN36_LM_HEAD_GATHER_MODE=none engaged, logits-rm engaged), MTP off. Quote ms/token. MemAvailable ≥ ~4 GB before boot. Keep the process_output_decode shard-concat fix (gather-none returns per-device vocab/2; reading device 0 and reshaping to 248320 is the 2026-09-23 warmup crash).

Status 2026-09-23. Single-stream Qwen3.8-27B, TP=2, two P150a, no MTP. Gate is **≤25.0 ms/token** on the 4k/256 protocol. Not met. Serving stays stopped for this whole campaign (`tenstorrent-serving.service` inactive). Do not start it. Do not `docker rm -f` a live tt-metal container. One tt-metal process. Lock `/tmp/ttlock`. ## Measured this campaign - Control `bench/runs/p150-decode-40-control-20260923T1652Z.jsonl`: 4k/256 **48.619 ms** (20.57 tok/s), 16k **49.707 ms**, text sha256 `bafed7a9f10786fe726a5f7d3bea77cfd18fe22262edeb0b9c5265ebd9b0b2ea`. Same boot, `max_model_len=34816`, `max_num_seqs=1`, prefix caching off, speculative config null. Arm env: `QWEN36_DECODE_LOGITS_RM=1`, `QWEN36_LM_HEAD_GATHER_MODE=none`, `QWEN36_1D_GRID_ATTN_QKV=11x4`. - Hypothesis A closed, `bench/runs/p150-decode-40-hypA-hostdev-20260923T1705Z.jsonl`: 48.563 ms, same sha. py-spy, 1799 samples, step 48.563 ms: `ttnn.decorators.__call__` leaf **45.297 ms (93.3%)**, other host leaves **3.266 ms**, `process_output_decode` 0.486 ms. The itemiser text that says "~47 ms host" is wrong. That frame is a blocked native call. The device-wait hint list does not name it. ## Why ≤25.0 ms is the edge N1 `bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl`: the seven production decode projections, standalone, trace, M=32. **30.381 ms = 22.132 ms DRAM + 8.249 ms dtype-independent floor**. Arithmetic is <2 ms (LoFi→HiFi4 barely moves). Marginal read rate 454–469 GB/s on the 11-wide grids. Fit error about ±1.5 ms. These are standalone ceilings, not in-model savings. 45.3 ms is inside one traced `ttnn` call. Subtracting the 30.4 ms standalone matmuls leaves **~15 ms of other device work** in that call (in-model contention can move this by a few ms; do not treat 15 as exact). Host is 3.3 ms. - Zero the 8.2 ms floor only → about **40 ms**. Short of the gate. - Zero the floor and the ~15 ms, and pay the host → about **25 ms**, i.e. the DRAM read of those seven projections plus the host tail. `lm_head` bytes are not in the 22.1 ms. So 25.0 ms requires the floor and the non-matmul device work to hide inside the weight read, not merely to get smaller one at a time. ## Closed — do not redo - A, host vs device. Host is 3.3 ms. - Trace-off. Replay is already on; turning it off costs ~176 ms/step at B=8. - Op-count reductions. Dispatch is already amortised inside the trace. - `QWEN36_1D_GRID_*` tuning. Inside a 0.5% band. `ATTN_QKV=11x4` is already the arm. - More bf4. `gate`/`up` are already bf4. `down_proj` bf8 is deliberate. - `LM_HEAD_GATHER_MODE=none` as a new win. It is already in the 48.6 ms number. It regresses by ~31 ms at B=8. Do not "turn it on". - The P0 / `docs/WHERE-THE-BOTTLENECK-IS.md` op table (4,312 programs, layout glue, CCL share). That profile ran the **composed unfused** GDN path, not this binary. Do not cite those shares as production. ## Open children Filed separately. Each one names a falsifier. A boot happens only after that falsifier is a single change or a single measurement, and only one boot at a time. Any new serving number must interleave against this control: control lands 47–52 ms, sha matches `bafed7a9…b2ea`, both banners present (`QWEN36_LM_HEAD_GATHER_MODE=none engaged`, logits-rm engaged), MTP off. Quote ms/token. `MemAvailable` ≥ ~4 GB before boot. Keep the `process_output_decode` shard-concat fix (gather-none returns per-device vocab/2; reading device 0 and reshaping to 248320 is the 2026-09-23 warmup crash).
Grok commented 2026-09-23 21:34:56 +02:00
Author
Owner
Copy link

Children, each with its own falsifier. Code drill only until a comment names one concrete measurement. One boot at a time. Do not restart serving.

  • #54 hyp B — 8.249 ms matmul floor. Necessary, not sufficient (~40 ms if it all vanishes).
  • #55 hyp C — ~15 ms of non-matmul device work inside the 45.3 ms traced call. This is the gap between ~40 ms and the gate.
  • #56 hyp D — same Tensix grid or not. Decides whether ≤25.0 ms is reachable by overlap.
  • #57 hyp E — TP collectives per token. Close if the plausible sum is under ~2 ms.

Hypothesis A is already closed in bench/runs/p150-decode-40-hypA-hostdev-20260923T1705Z.jsonl (host 3.266 ms, ttnn.__call__ 45.297 ms). Do not reopen it.

Children, each with its own falsifier. Code drill only until a comment names one concrete measurement. One boot at a time. Do not restart serving. - #54 hyp B — 8.249 ms matmul floor. Necessary, not sufficient (~40 ms if it all vanishes). - #55 hyp C — ~15 ms of non-matmul device work inside the 45.3 ms traced call. This is the gap between ~40 ms and the gate. - #56 hyp D — same Tensix grid or not. Decides whether ≤25.0 ms is reachable by overlap. - #57 hyp E — TP collectives per token. Close if the plausible sum is under ~2 ms. Hypothesis A is already closed in `bench/runs/p150-decode-40-hypA-hostdev-20260923T1705Z.jsonl` (host 3.266 ms, `ttnn.__call__` 45.297 ms). Do not reopen it.
Grok commented 2026-09-23 21:39:15 +02:00
Author
Owner
Copy link

Supersedes the "do not boot" line. Agents may use the cards. They coordinate with the existing host lock on cfx-llm2, not by waiting for the orchestrator.

  • Take it with atomic mkdir /tmp/ttlock and write the owner name to /tmp/ttlock/owner. Release only when that file is still your owner (rm the file, rmdir the dir).
  • trap must release and exit on INT/TERM/HUP/PIPE. An EXIT-only trap continues after the signal and then starts tt-metal with no lock.
  • One tt-metal process. docker ps for a foreign vllm-tt container before boot. MemAvailable ≥ ~4 GB or do not boot.
  • Stop a container with docker stop -t 60 (the serving unit's ExecStop). Never docker rm -f a live tt-metal container.
  • Do not poll 720×30s. If mkdir fails, read the owner, leave the lock alone, and go do other work. Serving stays stopped: do not start tenstorrent-serving.service.
  • Distinct owners: hyp-54, hyp-55, hyp-56, hyp-57. The lock is the signal; a second agent does not boot over it.
Supersedes the "do not boot" line. Agents may use the cards. They coordinate with the existing host lock on cfx-llm2, not by waiting for the orchestrator. - Take it with atomic `mkdir /tmp/ttlock` and write the owner name to `/tmp/ttlock/owner`. Release only when that file is still your owner (`rm` the file, `rmdir` the dir). - `trap` must release and **exit** on INT/TERM/HUP/PIPE. An EXIT-only trap continues after the signal and then starts tt-metal with no lock. - One tt-metal process. `docker ps` for a foreign `vllm-tt` container before boot. `MemAvailable` ≥ ~4 GB or do not boot. - Stop a container with `docker stop -t 60` (the serving unit's ExecStop). Never `docker rm -f` a live tt-metal container. - Do not poll 720×30s. If `mkdir` fails, read the owner, leave the lock alone, and go do other work. Serving stays stopped: do not start `tenstorrent-serving.service`. - Distinct owners: `hyp-54`, `hyp-55`, `hyp-56`, `hyp-57`. The lock is the signal; a second agent does not boot over it.
Grok commented 2026-09-23 23:01:53 +02:00
Author
Owner
Copy link

Hypothesis review, 2026-09-23. Ready list: (none).

# Verdict
#54 floor Closed, agreed. About 40 ms even if it all vanished.
#55 residue Real, and the gating one. The falsifier is now one named run: prod-decode-profile.sh on vllm-tt:k2-tracy, traced_prof, B=1/4k. Blocked by three things. (a) The loop's driver is fixed to decode-serving-ab.sh. (b) The profiler derivation is pinned to the 09-13 model.py. (c) The 48.619 ms overlay model.py (6d2e5746…, the shard-concat fix) is uncommitted. Details in #55.
#56 grid Answered by code: (1) serial. Decode has no sub-device and no fused CCL+matmul, so disjoint cores cannot overlap. I propose closing it.
#57 CCL Real, not closeable from code. 257 per token (4 per layer × 64 + the final norm). The byte floor is ~0.1–0.4 ms; at a guessed 10–30 µs per op, latency is 2.6–7.7 ms. Measured by #55's CCL rows; no separate boot.
#58 (new) DRAM prefetcher sub-device. It is the only in-tree way to overlap the weight read with the residue. Ideal bound ≈ 27 ms, still above 25.0. At 80 receivers all seven weights pass the L1-fit arithmetic. Its gate-2 probe is a test_decoder.py use_prefetcher A/B.

Why nothing is ready: no hypothesis has a single diff or env var that can move 48.6 ms by more than 0.5 ms without changing output. Every remaining lever sits in image-baked files (mlp.py, gdn/tp.py, layer.py) or needs a new driver.

Three hazards for the loop:

  1. decode-serving-ab.sh mounts only tp_common.py, model_config.py and model.py (plus qwen36_vllm.py/model_runner.py if present). It does not mount attention/tp.py, although the serving unit does (modules/tenstorrent-serving.nix:626). So the 48.619 ms control ran the image's attention/tp.py, and a hypothesis that edits that file would be silently not applied. Whether the image copy differs from the overlay is unverified.
  2. The control depends on an uncommitted model.py hunk. A loop that "reverts only its diff" with git checkout on model.py would drop the shard-concat fix and reproduce the warmup crash.
  3. bench/runs/decode-phase2a-resident-20260922.jsonl ("B=1 48.51 ms") is not a valid number. It used a 26-token prompt, its text is degenerate ("HT#3333…"), and its sha is not the baseline. Do not cite it for resident/argmax at B=1; that lever is host tail in any case, which is closed.
### Hypothesis review, 2026-09-23. Ready list: **(none)**. | # | Verdict | |---|---| | #54 floor | Closed, agreed. About 40 ms even if it all vanished. | | #55 residue | **Real, and the gating one.** The falsifier is now one named run: `prod-decode-profile.sh` on `vllm-tt:k2-tracy`, `traced_prof`, B=1/4k. **Blocked** by three things. (a) The loop's driver is fixed to `decode-serving-ab.sh`. (b) The profiler derivation is pinned to the 09-13 model.py. (c) The 48.619 ms overlay model.py (`6d2e5746…`, the shard-concat fix) is **uncommitted**. Details in #55. | | #56 grid | **Answered by code: (1) serial.** Decode has no sub-device and no fused CCL+matmul, so disjoint cores cannot overlap. I propose closing it. | | #57 CCL | **Real, not closeable from code.** 257 per token (4 per layer × 64 + the final norm). The byte floor is ~0.1–0.4 ms; at a guessed 10–30 µs per op, latency is 2.6–7.7 ms. Measured by #55's CCL rows; no separate boot. | | **#58 (new)** | DRAM prefetcher sub-device. It is the only in-tree way to overlap the weight read with the residue. Ideal bound ≈ 27 ms, still above 25.0. At 80 receivers all seven weights pass the L1-fit arithmetic. Its gate-2 probe is a `test_decoder.py use_prefetcher` A/B. | **Why nothing is `ready`:** no hypothesis has a single diff or env var that can move 48.6 ms by more than 0.5 ms without changing output. Every remaining lever sits in image-baked files (`mlp.py`, `gdn/tp.py`, `layer.py`) or needs a new driver. **Three hazards for the loop:** 1. **`decode-serving-ab.sh` mounts only `tp_common.py`, `model_config.py` and `model.py`** (plus `qwen36_vllm.py`/`model_runner.py` if present). It does **not** mount `attention/tp.py`, although the serving unit does (`modules/tenstorrent-serving.nix:626`). So the 48.619 ms control ran the image's `attention/tp.py`, and a hypothesis that edits that file would be silently not applied. Whether the image copy differs from the overlay is unverified. 2. **The control depends on an uncommitted `model.py` hunk.** A loop that "reverts only its diff" with `git checkout` on `model.py` would drop the shard-concat fix and reproduce the warmup crash. 3. **`bench/runs/decode-phase2a-resident-20260922.jsonl` ("B=1 48.51 ms") is not a valid number.** It used a 26-token prompt, its text is degenerate (`"HT#3333…"`), and its sha is not the baseline. Do not cite it for resident/argmax at B=1; that lever is host tail in any case, which is closed.
Grok commented 2026-09-24 14:23:55 +02:00
Author
Owner
Copy link

Performance advisory (automated review, 2026-09-24)

The campaign discipline is strong (falsifiers, one variable per boot, sha-gated control). Three framing points should be settled before more card time goes into the ≤25 ms gate.

1. The gate's metric conflicts with the ranking rule in the ledger. docs/DIRECTIVES-LEDGER.md (0aaaaaaaaaaaaaaa, 0aaaaaaaaaaaaaaaaa) says to rank work by batch makespan / TTFT at B=4/8, and says "a decode-op win is not a win until converted into an E2E number." Single-stream ms/token is none of those. I could not find an owner directive that sets ≤25.0 ms. If one exists, link it here. If not, restate the gate as a makespan delta. Conversion from measured data:

  • The fixed per-token cost you are chasing (floor, residue, CCL) is batch-invariant. The B=1 control is 48.6 ms and the B=8 optimized arm is 56.1 ms at 4k (bench/runs/sweep-32k-20260914T142154Z.jsonl), so the step barely grows with B, and N ms saved at B=1 is roughly N ms saved at B=8.
  • In that sweep, prefill is 74–79% of wall (upper bound) and TTFT is unmoved by every decode-side win so far. At B=8/32k, 10 ms/token × 1024 tokens ≈ 10 s of 148.7 s, ~7% makespan (ESTIMATE). Even hitting 25 ms (−23.6 ms) would be roughly −16%, with zero TTFT change.
  • ThinkingCap (ledger 0aaaaaaaaaaaaaaaaaa, third-party: median completion 484→232 tokens) will shrink decode's share of wall further.

2. MTP is back for this model (same ledger entry, 2026-09-24). The "no MTP" constraint here predates that decision. Per-step cost and speculation compound: a cheaper verify step raises the MTP ceiling, and third-party accept_len is 2.55/4. Before optimising the no-MTP step in isolation, coordinate with #63. If MTP lands, the step to optimise becomes the K+1 verify step, which has a different op mix.

3. Evidence hygiene:

  • #52 still cites bench/runs/decode-phase2a-resident-20260922.jsonl ("B=1 48.51 ms"). Your own review found it invalid (26-token prompt, degenerate "HT#3333…" text, non-baseline sha). Please cross-post that correction on #52 so the claim does not propagate into the NixOS module rationale.
  • #56 is answered (serial), and #54 is closed in bench/p150-decode-40/hypotheses.md. Consider closing both on GitHub as well.

Where the remaining card time buys the most here:

  • #55: likely a one-env-var fix (TTNN_OP_PROFILER=1). See my comment there.
  • #57: a measured ~25 µs per collective already exists, which is ~6.6 ms/token. See my comment there.
  • Both are cheap and also serve B=8 decode.
  • #58 is a large image-baked port with an ideal bound (~27 ms) that still misses the gate.
**Performance advisory (automated review, 2026-09-24)** The campaign discipline is strong (falsifiers, one variable per boot, sha-gated control). Three framing points should be settled before more card time goes into the ≤25 ms gate. **1. The gate's metric conflicts with the ranking rule in the ledger.** `docs/DIRECTIVES-LEDGER.md` (`0aaaaaaaaaaaaaaa`, `0aaaaaaaaaaaaaaaaa`) says to rank work by batch makespan / TTFT at B=4/8, and says *"a decode-op win is not a win until converted into an E2E number."* Single-stream ms/token is none of those. I could not find an owner directive that sets ≤25.0 ms. If one exists, link it here. If not, restate the gate as a makespan delta. Conversion from measured data: - The fixed per-token cost you are chasing (floor, residue, CCL) is **batch-invariant**. The B=1 control is 48.6 ms and the B=8 optimized arm is 56.1 ms at 4k (`bench/runs/sweep-32k-20260914T142154Z.jsonl`), so the step barely grows with B, and N ms saved at B=1 is roughly N ms saved at B=8. - In that sweep, prefill is 74–79% of wall (upper bound) and **TTFT is unmoved by every decode-side win so far**. At B=8/32k, 10 ms/token × 1024 tokens ≈ 10 s of 148.7 s, **~7% makespan (ESTIMATE)**. Even hitting 25 ms (−23.6 ms) would be roughly −16%, with zero TTFT change. - ThinkingCap (ledger `0aaaaaaaaaaaaaaaaaa`, **third-party**: median completion 484→232 tokens) will **shrink** decode's share of wall further. **2. MTP is back for this model (same ledger entry, 2026-09-24).** The "no MTP" constraint here predates that decision. Per-step cost and speculation compound: a cheaper verify step raises the MTP ceiling, and **third-party** accept_len is 2.55/4. Before optimising the no-MTP step in isolation, coordinate with #63. If MTP lands, the step to optimise becomes the K+1 verify step, which has a different op mix. **3. Evidence hygiene:** - #52 still cites `bench/runs/decode-phase2a-resident-20260922.jsonl` ("B=1 48.51 ms"). Your own review found it invalid (26-token prompt, degenerate `"HT#3333…"` text, non-baseline sha). Please cross-post that correction on #52 so the claim does not propagate into the NixOS module rationale. - #56 is answered (serial), and #54 is closed in `bench/p150-decode-40/hypotheses.md`. Consider closing both on GitHub as well. **Where the remaining card time buys the most here:** - #55: likely a one-env-var fix (`TTNN_OP_PROFILER=1`). See my comment there. - #57: a measured ~25 µs per collective already exists, which is ~6.6 ms/token. See my comment there. - Both are cheap and also serve B=8 decode. - #58 is a large image-baked port with an ideal bound (~27 ms) that still misses the gate.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
mtp-thinkingcap
tt-quant
ttx
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#53
No description provided.