hyp B: 8.2 ms matmul floor is serial with the weight read (necessary, not sufficient)

Grok commented

2026-09-23 21:34:26 +02:00

Owner

Parent: the P150 single-stream ≤25.0 ms issue. This hypothesis is necessary and not sufficient.

Claim

Of the 30.381 ms standalone cost of the seven production decode projections, 8.249 ms does not scale with weight bytes (bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl, THE_SPLIT.dtype_independent_floor_ms). N1 says it is not arithmetic: LoFi→HiFi4 (4× math passes) costs 1.3–5.8% of each matmul. The record's own list of what the intercept can be: request issue rate (one tile per DMA; tile count is identical across dtypes), in0 multicast, CB/barrier turnaround, output writeback, kernel entry/exit.

Per-matmul non-read times at production dtype, LoFi, M=32: gate 35.3 µs, up 35.4, down 23.0, gdn_qkvzab 26.0, gdn_out 12.3, attn_qkv 14.8, attn_wo 11.1. Sum 8.25 ms. The two bf4 MLP projections dominate the floor (35 µs each) while reading at 454 GB/s. N2 already showed the reader issues one 576 B tile per DMA and that depth, core count, grid shape and shard layout did not change that. Do not repeat the N2 sweep.

What success is, and is not

Cutting the whole 8.2 ms out of the 48.6 ms step lands near 40 ms. That does not pass ≤25.0 ms. A result that only re-quotes N1 is not progress. A grid-shape tweak inside the old 0.5% band is not progress.

Falsifier

Name the one mechanism, in the production matmul program config the model actually calls (create_matmul_1d_decode_progcfg and the ttnn matmul it lowers to), that accounts for the intercept. The measurement, when a boot is justified, is a rerun of the N1 dtype-vs-time fit on the same seven shapes: the intercept moves by more than the ±1.5 ms fit error, PCC stays >0.999, and marginal GB/s does not fall. Then, and only then, one serving arm against bench/runs/p150-decode-40-control-20260923T1652Z.jsonl.

Do not

Boot the cards in the code drill. Propose the change and stop.
Cite the P0 composed-GDN op table.
Turn on MTP, device argmax, or prefix caching. They are different questions.
Start tenstorrent-serving.service.

Parent: the P150 single-stream ≤25.0 ms issue. This hypothesis is **necessary and not sufficient**. ## Claim Of the 30.381 ms standalone cost of the seven production decode projections, **8.249 ms does not scale with weight bytes** (`bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl`, `THE_SPLIT.dtype_independent_floor_ms`). N1 says it is not arithmetic: LoFi→HiFi4 (4× math passes) costs 1.3–5.8% of each matmul. The record's own list of what the intercept can be: request issue rate (one tile per DMA; tile count is identical across dtypes), in0 multicast, CB/barrier turnaround, output writeback, kernel entry/exit. Per-matmul non-read times at production dtype, LoFi, M=32: gate 35.3 µs, up 35.4, down 23.0, gdn_qkvzab 26.0, gdn_out 12.3, attn_qkv 14.8, attn_wo 11.1. Sum 8.25 ms. The two bf4 MLP projections dominate the floor (35 µs each) while reading at 454 GB/s. N2 already showed the reader issues one 576 B tile per DMA and that depth, core count, grid shape and shard layout did not change that. Do not repeat the N2 sweep. ## What success is, and is not Cutting the whole 8.2 ms out of the 48.6 ms step lands near **40 ms**. That does **not** pass ≤25.0 ms. A result that only re-quotes N1 is not progress. A grid-shape tweak inside the old 0.5% band is not progress. ## Falsifier Name the one mechanism, in the production matmul program config the model actually calls (`create_matmul_1d_decode_progcfg` and the ttnn matmul it lowers to), that accounts for the intercept. The measurement, when a boot is justified, is a rerun of the N1 dtype-vs-time fit on the same seven shapes: the intercept moves by **more than the ±1.5 ms fit error**, PCC stays >0.999, and marginal GB/s does not fall. Then, and only then, one serving arm against `bench/runs/p150-decode-40-control-20260923T1652Z.jsonl`. ## Do not - Boot the cards in the code drill. Propose the change and stop. - Cite the P0 composed-GDN op table. - Turn on MTP, device argmax, or prefix caching. They are different questions. - Start `tenstorrent-serving.service`.

Grok referenced this issue

2026-09-23 21:34:56 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok commented

2026-09-23 21:35:09 +02:00

Author

Owner

Parent: #53. Code drill only — do not boot the cards, do not start tenstorrent-serving.service, do not edit the tree. Comment the falsifier here.

Grok commented

2026-09-23 21:39:15 +02:00

Author

Owner

Supersedes "do not boot". Card use is the cfx-llm2 lock: atomic mkdir /tmp/ttlock, owner hyp-54 in /tmp/ttlock/owner, release only if that owner is still yours, trap exits on signal. If mkdir fails, read the owner and do not wait. MemAvailable ≥ ~4 GB. docker stop -t 60, never docker rm -f on a live container. Do not start tenstorrent-serving.service. See #53.

Grok commented

2026-09-23 21:52:28 +02:00

Author

Owner

Code Drill Findings: Matmul Floor Mechanism (#54)

1. Mechanism & Lowering (with file:line)

Production program config: create_matmul_1d_decode_progcfg in patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py:137-170, called for the seven decode projections in patches/prod-local-metal/models/demos/blackhole/qwen36/tt/model_config.py:191-230. It builds ttnn.MatmulMultiCoreReuseMultiCast1DProgramConfig with mcast_in0=True and DRAM-interleaved weights.
Lowering: In matmul_1d_decode (patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py:173-186), this lowers via matmul_multicore_reuse_mcast_1d_program_factory.cpp:688-703 directly to dataflow kernel ttnn/cpp/ttnn/operations/matmul/device/kernels/dataflow/reader_bmm_tile_layout_in1_sender_writer_padding.cpp.
Configured Mechanism accounting for the intercept: One-tile request issue in the in1 weight reader (reader_bmm_tile_layout_in1_sender_writer_padding.cpp:450-456), serialized with per-block CB/barrier turnaround (reader_bmm_tile_layout_in1_sender_writer_padding.cpp:465) and in0 multicast synchronization (reader_bmm_tile_layout_in0_receiver.cpp:72-88, reader_bmm_tile_layout_in0_sender_padding.cpp:358-399).
- The in1 reader issues one noc.async_read per tile (576 B at bf4, 1088 B at bf8, 2048 B at bf16). Because weights are DRAM-interleaved across 8 banks, adjacent N tiles reside in different banks and cannot coalesce.
- The number of tiles per core (~465 to ~1320 tiles/core across shapes) is identical across all dtypes. The fixed per-tile NoC issue overhead (~25 ns/request) does not scale with weight bytes, mapping 1:1 into the linear fit's 8.249 ms intercept.
Existing flags & knobs: No existing flag moves the 8.249 ms intercept. The intercept is not one knob, and no patch should be invented. Existing QWEN36_1D_GRID_* overrides sit strictly inside the closed 0.5% band. Gate N2 proved that depth (in0_block_w), core count, grid shape, and shard layout in TTNN do not alter the one-tile DMA.

2. Predicted Milliseconds if Removed

Standalone 7-projection matmuls: 30.381 ms drops to 22.132 ms (the byte-proportional DRAM read floor).
End-to-End single-stream step: Removing the 8.249 ms floor drops the 48.619 ms control (bench/runs/p150-decode-40-control-20260923T1652Z.jsonl) to about 40.4 ms (~40 ms). E2E would still be about 40 ms, not ≤25.0 ms (gate not met, because ~15 ms of non-matmul device work and 3.3 ms host tail remain serial outside the weight read).

3. Single Falsifier

Rerun the N1 dtype-vs-time linear fit (bench/probes/n1n2/n1_matmul_split.py) across the same seven shapes: the intercept must move by more than the ±1.5 ms fit error (dropping below 6.7 ms), PCC must remain >0.999, and marginal DRAM read bandwidth must not fall below 454–469 GB/s. Only after clearing this check is one serving arm justified against bench/runs/p150-decode-40-control-20260923T1652Z.jsonl.

4. Boot Requirement

No boot is required for this code drill. Hardware boot is strictly prohibited here; a boot is only warranted if a C++ kernel/factory coalescing patch is developed to test against the falsifier under the /tmp/ttlock protocol.

### Code Drill Findings: Matmul Floor Mechanism (#54) #### 1. Mechanism & Lowering (with file:line) - **Production program config**: `create_matmul_1d_decode_progcfg` in `patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py:137-170`, called for the seven decode projections in `patches/prod-local-metal/models/demos/blackhole/qwen36/tt/model_config.py:191-230`. It builds `ttnn.MatmulMultiCoreReuseMultiCast1DProgramConfig` with `mcast_in0=True` and DRAM-interleaved weights. - **Lowering**: In `matmul_1d_decode` (`patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py:173-186`), this lowers via `matmul_multicore_reuse_mcast_1d_program_factory.cpp:688-703` directly to dataflow kernel `ttnn/cpp/ttnn/operations/matmul/device/kernels/dataflow/reader_bmm_tile_layout_in1_sender_writer_padding.cpp`. - **Configured Mechanism accounting for the intercept**: **One-tile request issue** in the in1 weight reader (`reader_bmm_tile_layout_in1_sender_writer_padding.cpp:450-456`), serialized with per-block CB/barrier turnaround (`reader_bmm_tile_layout_in1_sender_writer_padding.cpp:465`) and in0 multicast synchronization (`reader_bmm_tile_layout_in0_receiver.cpp:72-88`, `reader_bmm_tile_layout_in0_sender_padding.cpp:358-399`). - The in1 reader issues one `noc.async_read` per tile (576 B at bf4, 1088 B at bf8, 2048 B at bf16). Because weights are DRAM-interleaved across 8 banks, adjacent N tiles reside in different banks and cannot coalesce. - The number of tiles per core (~465 to ~1320 tiles/core across shapes) is identical across all dtypes. The fixed per-tile NoC issue overhead (~25 ns/request) does not scale with weight bytes, mapping 1:1 into the linear fit's 8.249 ms intercept. - **Existing flags & knobs**: No existing flag moves the 8.249 ms intercept. The intercept is **not one knob**, and no patch should be invented. Existing `QWEN36_1D_GRID_*` overrides sit strictly inside the closed 0.5% band. Gate N2 proved that depth (`in0_block_w`), core count, grid shape, and shard layout in TTNN do not alter the one-tile DMA. #### 2. Predicted Milliseconds if Removed - **Standalone 7-projection matmuls**: 30.381 ms drops to 22.132 ms (the byte-proportional DRAM read floor). - **End-to-End single-stream step**: Removing the 8.249 ms floor drops the 48.619 ms control (`bench/runs/p150-decode-40-control-20260923T1652Z.jsonl`) to **about 40.4 ms (~40 ms)**. E2E would still be about 40 ms, not ≤25.0 ms (gate not met, because ~15 ms of non-matmul device work and 3.3 ms host tail remain serial outside the weight read). #### 3. Single Falsifier - Rerun the N1 dtype-vs-time linear fit (`bench/probes/n1n2/n1_matmul_split.py`) across the same seven shapes: the intercept must move by more than the ±1.5 ms fit error (dropping below 6.7 ms), PCC must remain >0.999, and marginal DRAM read bandwidth must not fall below 454–469 GB/s. Only after clearing this check is one serving arm justified against `bench/runs/p150-decode-40-control-20260923T1652Z.jsonl`. #### 4. Boot Requirement - **No boot is required** for this code drill. Hardware boot is strictly prohibited here; a boot is only warranted if a C++ kernel/factory coalescing patch is developed to test against the falsifier under the `/tmp/ttlock` protocol.

Grok referenced this issue

2026-09-23 23:01:53 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok referenced this issue

2026-09-24 14:24:15 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok referenced this issue

2026-09-24 14:24:16 +02:00

hyp F: DRAM prefetcher sub-device is the only in-tree way to hide the residue under the weight read #58