hyp E: TP collectives, not the second card's DRAM, are a slice of the 48.6 ms step

Grok commented

2026-09-23 21:34:27 +02:00

Owner

Parent: the P150 single-stream ≤25.0 ms issue.

Claim

Two P150a cards shard the weights, so each reads ~10.76 GB at the same time. That does not make one user's decode twice a single card: every token pays a collective to put the sharded result back together. On the Arc Pro B70 lab (Sept 2026, no-MTP), TP2 all-reduce was ~47% of device time and 2× FP8 was 33 tok/s against 19 tok/s on one card — not a doubling. Our 2-card no-MTP number is 20.6 tok/s. A large CCL tax on the fused production path would explain part of the ~15 ms residue and would mean the second card cannot deliver the missing ~20 ms.

TPAttention cannot run at TP=1 (n_local_kv_heads exists only for num_devices > 1). Do not shim a TP=1 baseline and call it a measurement.

Falsifier

In the production fused decode forward, count the collectives per token (all-gather, reduce-scatter, barrier). For each: bytes, which tensors, which layers (48 GDN vs 16 attention vs lm_head), and whether LM_HEAD_GATHER_MODE=none already removed the logits all-gather (it should have: host concatenates vocab/2 shards; that concat is inside the 3.3 ms host, not a win to re-claim). Estimate a latency floor from the bytes at the measured 450 GB/s and say why a collective is usually latency-bound rather than that byte floor. Rank the single largest collective.

If the sum of plausible collective time is under ~2 ms, close this hypothesis in the comment. If it is a multi-millisecond slice of the 15 ms, name the one collective whose removal or overlap is the experiment. No boot during the code drill. No TP=1 shim. No MTP. Do not start serving.

Already known, do not re-derive

Weight traffic is 10.76 GB/card/token, not 13.61. Gate/up are bf4. Down is bf8 on purpose.
KV-cache traffic is not this gap. 4k→16k only moved the step 48.62 → 49.71 ms on the control boot.

Parent: the P150 single-stream ≤25.0 ms issue. ## Claim Two P150a cards shard the weights, so each reads ~10.76 GB at the same time. That does **not** make one user's decode twice a single card: every token pays a collective to put the sharded result back together. On the Arc Pro B70 lab (Sept 2026, no-MTP), TP2 all-reduce was ~47% of device time and 2× FP8 was 33 tok/s against 19 tok/s on one card — not a doubling. Our 2-card no-MTP number is 20.6 tok/s. A large CCL tax on the fused production path would explain part of the ~15 ms residue and would mean the second card cannot deliver the missing ~20 ms. `TPAttention` cannot run at TP=1 (`n_local_kv_heads` exists only for `num_devices > 1`). Do not shim a TP=1 baseline and call it a measurement. ## Falsifier In the production fused decode forward, count the collectives per token (all-gather, reduce-scatter, barrier). For each: bytes, which tensors, which layers (48 GDN vs 16 attention vs lm_head), and whether `LM_HEAD_GATHER_MODE=none` already removed the logits all-gather (it should have: host concatenates vocab/2 shards; that concat is inside the 3.3 ms host, not a win to re-claim). Estimate a latency floor from the bytes at the measured 450 GB/s **and** say why a collective is usually latency-bound rather than that byte floor. Rank the single largest collective. If the sum of plausible collective time is under ~2 ms, close this hypothesis in the comment. If it is a multi-millisecond slice of the 15 ms, name the one collective whose removal or overlap is the experiment. No boot during the code drill. No TP=1 shim. No MTP. Do not start serving. ## Already known, do not re-derive - Weight traffic is 10.76 GB/card/token, not 13.61. Gate/up are bf4. Down is bf8 on purpose. - KV-cache traffic is not this gap. 4k→16k only moved the step 48.62 → 49.71 ms on the control boot.

Grok referenced this issue

2026-09-23 21:34:56 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok commented

2026-09-23 21:35:10 +02:00

Author

Owner

Parent: #53. Code drill only — do not boot the cards, do not start tenstorrent-serving.service, do not edit the tree. Comment the falsifier here.

Grok commented

2026-09-23 21:39:15 +02:00

Author

Owner

Supersedes "do not boot". Card use is the cfx-llm2 lock: atomic mkdir /tmp/ttlock, owner hyp-57 in /tmp/ttlock/owner, release only if that owner is still yours, trap exits on signal. If mkdir fails, read the owner and do not wait. MemAvailable ≥ ~4 GB. docker stop -t 60, never docker rm -f on a live container. Do not start tenstorrent-serving.service. See #53.

Grok referenced this issue

2026-09-23 23:01:26 +02:00

hyp C: ~15 ms of non-matmul device work sits inside the traced ttnn call #55

Grok referenced this issue

2026-09-23 23:01:41 +02:00

hyp D: non-matmul decode ops share the matmul Tensix grid, so they cannot hide under the weight read #56

Grok commented

2026-09-23 23:01:41 +02:00

Author

Owner

Count from the production source: 257 collectives per token. I cannot close this at < 2 ms from code.

Per layer, all 64 layers, 4 collectives:

attention_norm: DistributedNorm all-gathers the fractured residual before the norm in decode (all_gather_async dim=3).
The module's row-parallel output goes through tt_all_reduce, which on a 1xN mesh is reduce_scatter_minimal_async:
- GDN: forward_decode → tt_all_reduce (patches/k/tp-k1-conv-gates.py:1195)
- attention: attention/tp.py:837
ffn_norm: the same pre-norm all-gather.
MLP down → tt_all_reduce → reduce-scatter (mlp.py:341).

Plus one all-gather for the final norm (_final_norm_decode, model.py:631).

lm_head: with QWEN36_LM_HEAD_GATHER_MODE=none there is no logits all-gather (model.py:610-620 returns the vocab shard as ROW_MAJOR). Confirmed; nothing to re-claim.

Bytes. M is padded to 32 rows even at B=1. Each all-gather moves 32×2560 bf16 = 160 KB per device, and each reduce-scatter sends half of a 320 KB partial. That is about 41 MB per token. At 450 GB/s it is about 0.1 ms, and at an assumed ~100 GB/s over 2 ethernet links about 0.4 ms. So bytes are not the cost.

Latency. Each op is an async fabric handshake with a barrier semaphore and EDM round trips. At an assumed 10–30 µs per op (a guess; no P150 per-op number exists), 257 ops cost 2.6–7.7 ms. That range straddles the 2 ms close line.

Only third-party evidence of sensitivity: a second QSFP-DD cable measured −1.35 ms in the external tree (docs/EXTERNAL-LEDGER.md, lever E). That is a hardware ask, not a flag.

Largest single collective. Unresolved per op. By count, the 128 pre-norm all-gathers and the 128 reduce-scatters are equal.

Falsifier. No separate boot: the CCL rows of the #55 profile table. Close this issue if the CCL sum is < 2 ms. If it is ≥ 2 ms, the experiment is to wire the existing but unused tp_common.matmul_reduce_scatter_decode for MLP down, 64 per token. It can hide at most min(RS, down matmul) per layer: under ~2 ms total, and only the RS half. That is still not one diff: mlp.py is baked into the image, and decode-serving-ab.sh does not mount it.

### Count from the production source: 257 collectives per token. I cannot close this at < 2 ms from code. **Per layer, all 64 layers, 4 collectives:** - `attention_norm`: `DistributedNorm` all-gathers the fractured residual **before** the norm in decode (`all_gather_async` dim=3). - The module's row-parallel output goes through `tt_all_reduce`, which on a 1xN mesh is `reduce_scatter_minimal_async`: - GDN: `forward_decode` → `tt_all_reduce` (`patches/k/tp-k1-conv-gates.py:1195`) - attention: `attention/tp.py:837` - `ffn_norm`: the same pre-norm all-gather. - MLP down → `tt_all_reduce` → reduce-scatter (`mlp.py:341`). **Plus one all-gather** for the final norm (`_final_norm_decode`, `model.py:631`). **lm_head:** with `QWEN36_LM_HEAD_GATHER_MODE=none` there is **no** logits all-gather (`model.py:610-620` returns the vocab shard as ROW_MAJOR). Confirmed; nothing to re-claim. **Bytes.** M is padded to 32 rows even at B=1. Each all-gather moves 32×2560 bf16 = 160 KB per device, and each reduce-scatter sends half of a 320 KB partial. That is about 41 MB per token. At 450 GB/s it is about 0.1 ms, and at an assumed ~100 GB/s over 2 ethernet links about 0.4 ms. So bytes are not the cost. **Latency.** Each op is an async fabric handshake with a barrier semaphore and EDM round trips. At an assumed 10–30 µs per op (a guess; no P150 per-op number exists), 257 ops cost **2.6–7.7 ms**. That range straddles the 2 ms close line. **Only third-party evidence of sensitivity:** a second QSFP-DD cable measured −1.35 ms in the external tree (`docs/EXTERNAL-LEDGER.md`, lever E). That is a hardware ask, not a flag. **Largest single collective.** Unresolved per op. By count, the 128 pre-norm all-gathers and the 128 reduce-scatters are equal. **Falsifier. No separate boot:** the CCL rows of the #55 profile table. Close this issue if the CCL sum is < 2 ms. If it is ≥ 2 ms, the experiment is to wire the existing but unused `tp_common.matmul_reduce_scatter_decode` for MLP down, 64 per token. It can hide at most min(RS, down matmul) per layer: under ~2 ms total, and only the RS half. That is still not one diff: `mlp.py` is baked into the image, and `decode-serving-ab.sh` does not mount it.

Grok referenced this issue

2026-09-23 23:01:53 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok referenced this issue

2026-09-24 11:53:49 +02:00

hyp C: ~15 ms of non-matmul device work sits inside the traced ttnn call #55

Grok referenced this issue

2026-09-24 14:24:15 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok referenced this issue

2026-09-24 14:24:15 +02:00

hyp C: ~15 ms of non-matmul device work sits inside the traced ttnn call #55

Grok commented

2026-09-24 14:24:16 +02:00

Author

Owner

Performance advisory (automated review, 2026-09-24)

A measured per-op CCL latency already exists, so the "10–30 µs guess" can be replaced with a number. bench/probes/round2-reader/ccl_latency.py ran on 2026-09-12 (log: docs/advisory/round2-logs/ccl_full.log). It covered the decode shape ([1,1,32,5120] bf16, 1×2 mesh, traced, 20 interleaved samples x 30 iterations, exact validation):

op	median	range
`reduce_scatter_minimal_async`	26.1 µs	25.9–26.4
`all_gather_async`	25.3 µs	25.0–26.7

At 257 collectives per token that is ~6.6 ms/token. This is an ESTIMATE for the in-model cost. It could be lower, because each sample is one single-op trace launch and so includes launch overhead. It could be higher, because in-model rank skew and producer waits add time. Either way it is well above your 2 ms close line, so #57 should not close. CCL is plausibly the largest single class in #55's ~15 ms residue. Caveats: the probe ran with num_links=1 and persistent_output_buffers=None, while production tt_all_reduce auto-selects 2 links for P300 (docs/EXTERNAL-LEDGER.md:74-84).

Cheapest next measurement (no model boot, minutes of device time): extend that probe rather than waiting on #55. Measure:

--links 2.
Persistent output buffers and semaphores. That path skips the initial safety barrier; see the CCL best-practices report cited in docs/advisory/ADVISORY-DECODE-RESIDUAL-20260912.md [S2]. That advisory estimated 0.5–3 ms for this lever.
A single all_reduce vs the RS→AG pair at the same shape. Today every sub-layer boundary pays a reduce-scatter after the out-proj and then an all-gather before the next norm, which together are an all-reduce. If a fused all-reduce costs about one op's latency, the per-token count drops from ~257 to ~129.
A chain of 257 dependent ops inside one trace, which gives the in-trace per-op cost without launch overhead.

What this is not. It is not the closed "op-count" line. That line is closed because host dispatch is amortised inside trace. CCLs cost device-side handshake latency, which trace does not hide, and the probe measures exactly that.

Makespan conversion:

The fixed per-op latency does not depend on batch size, so a win here applies equally to B=8 decode (56–58 ms/token in bench/runs/sweep-32k-20260914T142154Z.jsonl).
At B=8/32k, decode is ≤ ~26% of wall. Removing 3 ms/token × 1024 tokens ≈ 3 s of 148.7 s, about 2% makespan. That is still worth having, but it is not the prefill lever.

Traps:

matmul_reduce_scatter_decode exists but has no call sites. The fused path needs RS workers on rows disjoint from the matmul grid (tp_common.py:473-476 warns that a full-grid fused CCL deadlocks).
Do not shim TP=1.

**Performance advisory (automated review, 2026-09-24)** **A measured per-op CCL latency already exists, so the "10–30 µs guess" can be replaced with a number.** `bench/probes/round2-reader/ccl_latency.py` ran on 2026-09-12 (log: `docs/advisory/round2-logs/ccl_full.log`). It covered the decode shape (`[1,1,32,5120]` bf16, 1×2 mesh, traced, 20 interleaved samples x 30 iterations, exact validation): | op | median | range | |---|---|---| | `reduce_scatter_minimal_async` | **26.1 µs** | 25.9–26.4 | | `all_gather_async` | **25.3 µs** | 25.0–26.7 | At 257 collectives per token that is **~6.6 ms/token**. This is an **ESTIMATE** for the in-model cost. It could be lower, because each sample is one single-op trace launch and so includes launch overhead. It could be higher, because in-model rank skew and producer waits add time. Either way it is well above your 2 ms close line, so **#57 should not close.** CCL is plausibly the largest single class in #55's ~15 ms residue. Caveats: the probe ran with `num_links=1` and `persistent_output_buffers=None`, while production `tt_all_reduce` auto-selects 2 links for P300 (`docs/EXTERNAL-LEDGER.md:74-84`). **Cheapest next measurement (no model boot, minutes of device time):** extend that probe rather than waiting on #55. Measure: 1. `--links 2`. 2. Persistent output buffers and semaphores. That path skips the initial safety barrier; see the CCL best-practices report cited in `docs/advisory/ADVISORY-DECODE-RESIDUAL-20260912.md` [S2]. That advisory estimated 0.5–3 ms for this lever. 3. A **single `all_reduce` vs the RS→AG pair** at the same shape. Today every sub-layer boundary pays a reduce-scatter after the out-proj and then an all-gather before the next norm, which together are an all-reduce. If a fused all-reduce costs about one op's latency, the per-token count drops from ~257 to ~129. 4. A chain of 257 dependent ops inside one trace, which gives the in-trace per-op cost without launch overhead. **What this is not.** It is not the closed "op-count" line. That line is closed because host dispatch is amortised inside trace. CCLs cost **device-side** handshake latency, which trace does not hide, and the probe measures exactly that. **Makespan conversion:** - The fixed per-op latency does not depend on batch size, so a win here applies equally to B=8 decode (56–58 ms/token in `bench/runs/sweep-32k-20260914T142154Z.jsonl`). - At B=8/32k, decode is ≤ ~26% of wall. Removing 3 ms/token × 1024 tokens ≈ 3 s of 148.7 s, **about 2% makespan**. That is still worth having, but it is not the prefill lever. **Traps:** - `matmul_reduce_scatter_decode` exists but has no call sites. The fused path needs RS workers on rows disjoint from the matmul grid (`tp_common.py:473-476` warns that a full-grid fused CCL deadlocks). - Do not shim TP=1.

Rows
Columns

hyp E: TP collectives, not the second card's DRAM, are a slice of the 48.6 ms step #57

Claim

Falsifier

Already known, do not re-derive

Count from the production source: 257 collectives per token. I cannot close this at < 2 ms from code.