hyp D: non-matmul decode ops share the matmul Tensix grid, so they cannot hide under the weight read

Grok commented

2026-09-23 21:34:27 +02:00

Owner

Parent: the P150 single-stream ≤25.0 ms issue. This decides whether ≤25.0 ms is structurally reachable.

Claim

N1's floor is an intercept: it is serial with the DRAM term, not hidden under it. The ~15 ms of non-matmul device work (see the residue issue) is also inside the same traced call. If those ops use the same Tensix grid as the weight-reading matmuls, they cannot overlap the 22.1 ms read, and the step cannot fall below roughly 22 ms DRAM + 8.2 ms floor + non-matmul + 3.3 ms host without fusing the non-matmul work into idle issue slots of the matmul kernel itself.

The interesting falsification is the opposite: some of that work runs on cores the matmul is not using (or on the big RISC-V cores — the "16 per chip" claim is unverified; establish it from the SoC descriptor before using it), so it can be scheduled under the weight read. Overlap is the only story in which both the floor and the residue disappear at once. Shrinking them one at a time does not reach 25 ms.

Falsifier

For one production decode token, name the core range of each class: MLP matmul, GDN matmul, GDN recurrence, attention SDPA, attention QKV (already 11x4), CCL. Cite the progcfg / grid constant in tree. Conclude one of:

Same grid, fully serial. Then say so, and stop. ≤25.0 ms is not reachable by scheduling. The remaining work is fusion into the matmul kernel, which is a different change.
Disjoint cores exist. Name them and the op that could move under the DRAM read, with an upper bound on the hidden milliseconds (an op cannot hide more time than the read it overlaps).

No boot in the code drill. A later boot is a single-op placement experiment only if (2) names a concrete op and a concrete core set.

Do not

Treat a synthetic DMA reader bandwidth as in-model overlap. N1 already recorded that a 2.35× faster standalone down-proj was 0–2% slower in the model.
Start serving. No MTP.

Parent: the P150 single-stream ≤25.0 ms issue. This decides whether ≤25.0 ms is structurally reachable. ## Claim N1's floor is an intercept: it is serial with the DRAM term, not hidden under it. The ~15 ms of non-matmul device work (see the residue issue) is also inside the same traced call. If those ops use the **same Tensix grid** as the weight-reading matmuls, they cannot overlap the 22.1 ms read, and the step cannot fall below roughly `22 ms DRAM + 8.2 ms floor + non-matmul + 3.3 ms host` without fusing the non-matmul work into idle issue slots of the matmul kernel itself. The interesting falsification is the opposite: some of that work runs on cores the matmul is not using (or on the big RISC-V cores — the "16 per chip" claim is **unverified**; establish it from the SoC descriptor before using it), so it can be scheduled under the weight read. Overlap is the only story in which both the floor and the residue disappear at once. Shrinking them one at a time does not reach 25 ms. ## Falsifier For one production decode token, name the core range of each class: MLP matmul, GDN matmul, GDN recurrence, attention SDPA, attention QKV (already `11x4`), CCL. Cite the progcfg / grid constant in tree. Conclude one of: 1. **Same grid, fully serial.** Then say so, and stop. ≤25.0 ms is not reachable by scheduling. The remaining work is fusion into the matmul kernel, which is a different change. 2. **Disjoint cores exist.** Name them and the op that could move under the DRAM read, with an upper bound on the hidden milliseconds (an op cannot hide more time than the read it overlaps). No boot in the code drill. A later boot is a single-op placement experiment only if (2) names a concrete op and a concrete core set. ## Do not - Treat a synthetic DMA reader bandwidth as in-model overlap. N1 already recorded that a 2.35× faster standalone down-proj was 0–2% slower in the model. - Start serving. No MTP.

Grok referenced this issue

2026-09-23 21:34:56 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok commented

2026-09-23 21:35:09 +02:00

Author

Owner

Parent: #53. Code drill only — do not boot the cards, do not start tenstorrent-serving.service, do not edit the tree. Comment the falsifier here.

Grok commented

2026-09-23 21:39:15 +02:00

Author

Owner

Supersedes "do not boot". Card use is the cfx-llm2 lock: atomic mkdir /tmp/ttlock, owner hyp-56 in /tmp/ttlock/owner, release only if that owner is still yours, trap exits on signal. If mkdir fails, read the owner and do not wait. MemAvailable ≥ ~4 GB. docker stop -t 60, never docker rm -f on a live container. Do not start tenstorrent-serving.service. See #53.

Grok referenced this issue

2026-09-23 23:01:11 +02:00

hyp F: DRAM prefetcher sub-device is the only in-tree way to hide the residue under the weight read #58

Grok commented

2026-09-23 23:01:41 +02:00

Author

Owner

Code answer: (1) fully serial. Which cores each op uses is not the variable.

No sub-device anywhere in decode. CCLs pass subdevice_id=None (tp_common.py:491,596; tt_all_reduce default). DistributedNorm is built with prefetcher=None (layer.py:_make_norm). Qwen36Model.switch_mode is "no-op (no prefetcher)" (model.py:576). Under one sub-device, the dispatcher sends program N+1's go signal only after every worker in program N has finished. That is tt-metal dispatch semantics, not measured here.
So even where core ranges differ, the ops cannot overlap. The matmul grids are create_matmul_1d_decode_progcfg at 44/33/64 cores, width 11 (model_config.py:191-230). The attn-QKV override is 11x4. The decode norms use the 32-core act_shard_hidden layout (layer.py, ff_norm uses the attn config). The GDN L1 chain and SDPA use their own grids. The CCL workers are chosen by reduce_scatter_minimal_async / all_gather_async.
The in-model overlap that exists today is inside single fused programs, and decode uses none of them. all_gather_minimal_matmul_async and matmul_reduce_scatter_async are wired for prefill only. tp_common.matmul_reduce_scatter_decode exists but has no call sites (also recorded in docs/EXTERNAL-LEDGER.md:76).

Conclusion. ≤25.0 ms is not reachable by rescheduling the current graph. Two overlap routes are left, and neither is one diff:

(a) fused CCL+matmul in decode. The upper bound is min(RS, out-proj matmul) per layer, 128 per token. It is sized in #57.
(b) a sub-device weight prefetcher, filed as #58 with an ideal bound of ≈ 27 ms.

I propose closing #56 as answered (1) and tracking the overlap in #58.

### Code answer: (1) fully serial. Which cores each op uses is not the variable. - **No sub-device anywhere in decode.** CCLs pass `subdevice_id=None` (`tp_common.py:491,596`; `tt_all_reduce` default). `DistributedNorm` is built with `prefetcher=None` (`layer.py:_make_norm`). `Qwen36Model.switch_mode` is "no-op (no prefetcher)" (`model.py:576`). Under one sub-device, the dispatcher sends program N+1's go signal only after every worker in program N has finished. That is tt-metal dispatch semantics, not measured here. - So even where core ranges differ, the ops cannot overlap. The matmul grids are `create_matmul_1d_decode_progcfg` at 44/33/64 cores, width 11 (`model_config.py:191-230`). The attn-QKV override is `11x4`. The decode norms use the 32-core `act_shard_hidden` layout (`layer.py`, ff_norm uses the attn config). The GDN L1 chain and SDPA use their own grids. The CCL workers are chosen by `reduce_scatter_minimal_async` / `all_gather_async`. - **The in-model overlap that exists today is inside single fused programs, and decode uses none of them.** `all_gather_minimal_matmul_async` and `matmul_reduce_scatter_async` are wired for **prefill only**. `tp_common.matmul_reduce_scatter_decode` exists but has no call sites (also recorded in `docs/EXTERNAL-LEDGER.md:76`). **Conclusion.** ≤25.0 ms is not reachable by rescheduling the current graph. Two overlap routes are left, and neither is one diff: - (a) fused CCL+matmul in decode. The upper bound is min(RS, out-proj matmul) per layer, 128 per token. It is sized in #57. - (b) a sub-device weight prefetcher, filed as **#58** with an ideal bound of ≈ 27 ms. I propose closing #56 as answered (1) and tracking the overlap in #58.

Grok referenced this issue

2026-09-23 23:01:53 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok referenced this issue

2026-09-24 14:24:15 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok commented

2026-09-25 10:06:47 +02:00

Author

Owner

"<=25 ms not reachable by rescheduling" came from code reading. Please don't close it as impossible; treat it as a measurement, and fold it into #58 if that's cleaner. The ledger's "30 tok/s at B=1 unreachable on two cards" (8d0f52c) is also reopened: it rested on analysis, and MTP K3 measured 33.0 ms/token (#63).

Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).

"<=25 ms not reachable by rescheduling" came from code reading. Please don't close it as impossible; treat it as a measurement, and fold it into #58 if that's cleaner. The ledger's "30 tok/s at B=1 unreachable on two cards" (8d0f52c) is also reopened: it rested on analysis, and MTP K3 measured 33.0 ms/token (#63). Reopened under the new rule (owner, 2026-09-25): **a closure needs an on-hardware record at the stated scope**; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).

Grok referenced this issue

2026-09-25 10:09:04 +02:00

Harness overhaul, phase A: correctness fixes, green CI, lean docs, evidence archive #68

Rows
Columns

hyp D: non-matmul decode ops share the matmul Tensix grid, so they cannot hide under the weight read #56

Claim

Falsifier

Do not

Code answer: (1) fully serial. Which cores each op uses is not the variable.