hyp F: DRAM prefetcher sub-device is the only in-tree way to hide the residue under the weight read #58
Labels
No labels
human-approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack#58
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent: #53. Follows from the #56 code answer.
Claim
The decode step is serial because nothing in it uses a sub-device. Every CCL call passes
subdevice_id=None(tp_common.py:491,596),DistributedNormgetsprefetcher=None, andQwen36Model.switch_modesays "no-op (no prefetcher)" (model.py:576). Under one sub-device, tt-metal starts program N+1 only after program N finishes on every core, whichever cores each one uses. That is dispatch semantics, not something measured here. So the 22.1 ms weight read and the ~15 ms residue (#55) add up. They cannot overlap.The in-tree way to overlap them is the Blackhole DRAM prefetcher,
models/tt_transformers/tt/prefetcher.py. Sender cores next to the DRAM banks sit on their own sub-device and stream the next layer's weights into a global circular buffer on the receiver (matmul) cores. Meanwhile norms, CCLs, GDN and SDPA run on the worker sub-device.VERIFIED_MODEL_CONFIGSincludes Qwen3-32B, which has dim 5120, the same as ours. third-party: we have never measured the prefetcher on P150.Predicted ms
Only an ideal bound, not an expected value. If the prefetcher hid everything, the step would be about max(weight read ≈ 23.6 ms including lm_head, worker-side ≈ 8.2 floor + ~15 residue) + 3.3 host ≈ 27 ms. That is still above 25.0 ms. The realistic gain is unknown until gate 2 runs.
Gate 1 (code only, no boot): does each weight fit in L1 at TP=2?
I applied the formula in
is_prefetcher_supported(850 KB per receiver at num_devices=2) to the production per-device shapes from N1 at production dtypes:down(8704×5120 bf8) andgdn_qkvzab(5120×8240 bf8) do not fit, at 1.04 MB each. The other five fit.prefetcher_config.yaml): all seven fit. The largest is 696 KB.So L1 capacity does not rule this out on paper. Still unchecked: double-buffering headroom next to the GDN decode L1 chain (
_L1conv→recurrence→norm) and whether the ring matmul (gather_in0) accepts bf4 in1.Gate 2: the falsifier, one probe boot
A standalone run, not a serving arm:
models/tt_transformers/tests/test_decoder.pyalready has ause_prefetcherparameter. Run decode at Qwen3-32B dims on the 1x2 P150 mesh with the trace on,use_prefetcherTrue vs False, and time one decoder layer per replay (a small timing wrapper is needed; the test only checks PCC).Why this is not
readyA port means ring matmuls on all seven projections plus sub-device setup, in
mlp.py,gdn/tp.pyandlayer.py. Those files are baked intovllm-tt:k2, andbench/ab/decode-serving-ab.shdoes not mount them. So this is not one diff. One tt-metal process; no TP=1; no MTP.Performance advisory (automated review, 2026-09-24)
Using the prefetcher as the overlap mechanism is sound. Five points should shape gate 2 and the port.
tt_transformers/tests/test_decoder.pybuildsPrefetcher(..., num_tensors=5, num_layers=1), withn_layers = 1. With one layer there is no "next layer's weights stream while this layer's norms, CCL and SDPA run", and that cross-layer overlap is the whole ≈27 ms story. Usedemo/simple_text_demo.py --use_prefetcher True --num_layers N(N ≥ 4) instead. This is the path upstream's owntests/test_device_perf.pydrives. Run it on/off on aVERIFIED_MODEL_CONFIGSmodel at TP=2 and read the demo's decode ms/token. Qwen3-32B (dim 5120) is the closest match. A smaller verified model, e.g. Llama-3.1-8B, answers "does BH prefetch overlap at all on P150 TP=2" with far less checkpoint and host-RAM pain; mind the ~5.5 GB probe RAM limit on cfx-llm2.docs/PREFILL-AUDIT-2026-09-12.md§4, where a probe predicted -9% and the model delivered 0 to +2%. Treat any layer-level gain as an upper bound.generator.py:540-586saysswitch_mode(Mode.PREFILL)"is not functional" with a prefetcher, and that prefetcher models skip the hoisted decode-trace setup. In our workload every arriving agent triggers a prefill→decode switch, and prefill owns TTFT. The cost of re-initialising the sub-devices on every switch has to be measured, and charged against the decode gain, before a port is justified. Add a gate 2b: time N prefill↔decode switches with the prefetcher on.gdn_qkvzab(1.04 MB/receiver at 64 receivers) and the GDN L1 chain (_L1conv→recurrence→norm) both compete for the same receiver L1 as the global CB. The 80-receiver fit arithmetic is necessary but not sufficient, and the double-buffering headroom is still unchecked, as you note.Makespan framing (see #53): the fixed per-token cost is batch-invariant, so a hypothetical 10 ms/token at B=8/32k is ~10 s of 148.7 s, about 7% makespan (ESTIMATE). That is real, but it is a large, multi-file, image-baked port (
mlp.py,gdn/tp.py,layer.py). It ranks below the warm-prefix prefill tail (#50) and below prefix deployment (#48), which move TTFT, the thing decode-side work has never moved (bench/runs/sweep-32k-20260914T142154Z.jsonl).