docs: scout inventory of MTP speculative decoding on Tenstorrent Blackhole #22
No reviewers
Labels
No labels
human-approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack!22
Loading…
Reference in a new issue
No description provided.
Delete branch "mtp-scout"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Comprehensive scout across HuggingFace, GitHub, and academic literature for existing Multi-Token Prediction (MTP) speculative decoding and Gated DeltaNet (GDN) state management on Tenstorrent Blackhole, mapped against the charter in
docs/GOAL-30TPS.md.Key Breakthroughs Uncovered:
mtp.*weights, and runs traced verify and commit. Measured 50.45 tok/s at 4k prompt and 51.31 tok/s at 16k prompt on hardware (exceeding our charter's >= 30 tok/s target).fused_recurrent_gated_delta_rulewith per-token state buffers and traced in-placecommit_verify_slot_opson device.sjettTT/tt-qwen-3.8-flash-next) implements an alternative snapshot-free chunk replay withcommitted_maskthat zeros beta/g past the accepted prefix.tt-hous/ornith-1.0-35b-p150resolved the TTNN default scheduler bug causing 48-layer recurrent matmuls to starve onto only 4 cores, alongside depthwise conv and layout optimizations.anyin233/qwen36-ttmeasured that graph tracing eliminates host round-trips for a 2.5x speedup (4.88 ms -> 1.94 ms/layer).Document committed at
docs/MTP-SCOUT-20260911.md.🤖 Generated with Claude Code
Result: {"status":"keep","gdn_commit_ops_shadow_k7":8,"gdn_commit_steps_shadow_commit_k7":8,"gdn_commit_copies_shadow_commit_k7":0,"gdn_commit_mb_shadow_commit_k7":0,"gdn_commit_copies_snapshot_k7":1,"gdn_commit_copies_recompute_k7":1,"gdn_commit_steps_snapshot_k7":15,"gdn_commit_steps_recompute_k7":15,"gdn_commit_state_consistent":1,"gdn_layer_bytes":1073152}Drives the production T=1 GDN decode primitive (recurrent_gated_delta_rule_decode_ttnn) through our StateTransaction protocol at TP=2. For K in {1,3,7} and every acceptance length a in [0,K], each of the three commit mechanisms (snapshot / shadow_commit / recompute) must commit the GDN recurrent state matching an autoregressive oracle replaying the identical token sequence one decode step at a time; a rejected-verify (abort) plus a later accept is also covered. Uses the 9B/27B GDN state shape [B, Nv, Dk, Dv] fp32 at B=1, Nv=8 (reduced from 32 to stay under the GDN decode kernel's compute-grid cap), Dk=Dv=128. The conv state is tracked as a per-token q shift register (the primitive covers the recurrent state, which is the D8.1 rollback core; the conv path is soft-checked). No checkpoint weights needed (deterministic per-token inputs). Registered in scripts/dspark-harness.py PROBES (name gdn-transaction -> bench/probes/dspark-gdn-transaction-tp.py).Drives the production T=1 GDN decode primitive (recurrent_gated_delta_rule_decode_ttnn) through our StateTransaction protocol on a 2-device mesh. For K in {1,3,7} and every acceptance length a in [0,K], all three commit mechanisms (snapshot / shadow_commit / recompute) commit the GDN recurrent state bit-identical to the autoregressive oracle that re-advances the pre-verify state one decode step at a time (recPCC=1.0000 for all 39 cells). A rejected verify (abort) leaves the live state exactly unchanged (torch.equal, after driving it to a non-trivial state), and a later commit after that rejection still matches the oracle advanced from the unchanged live state. The conv history is tracked as a per-token q shift register (a proxy); the primary D8.1 validation is the recurrent-state equality. performance: null (correctness only). Record: bench/runs/dspark-20260908T041702Z-6a5ffeb8.jsonl.First hardware run of bench/probes/dspark-verify-tp.py. K in {1,3,7}, paged and unpaged, worst PCC 0.999996, VERDICT: PASS on the 2x P150a. It had never run on cards, and could not have. The probe put an implicitly- concatenated string INSIDE an f-string expression, which parses only on Python 3.12+ (PEP 701). The nix runtime shell is 3.14, so it parsed there and every prior run went through the simulator; the vllm-tt container is Python 3.10, where it was a SyntaxError before the first line executed. That single incompatibility is why this probe's whole recorded history is simulator runs. Worse, the record it writes hard-coded "chips": "P150a x2 (simulator)". Running it on hardware would have produced a false record claiming the opposite of what happened. The label is now derived from TT_METAL_SIMULATOR, which is what actually selects the backend, so neither direction can be mislabelled; the timing-claim caveat is likewise conditioned on the real runtime. What this is NOT: it is target-only packed-SDPA kernel parity -- the packed K+1 call reproducing K+1 sequential single-position calls. There is no argmax, no accept decision, no GDN state and no commit anywhere in it. It does not close the shared-verifier row, and the record says so; gate_complete=false, performance=null. Next: the single decision owner (select_prefix -> Decision), folding in the EOS semantics where the tip emits no correction token but we currently would. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The owner asked for true uplift at 4k/8k/16k over the ~15 tok/s no-drafter baseline. Half of that is deliverable today and half is not, so this commit delivers the half that is and says clearly why the other half isn't. Not deliverable: there is no speculative decoding in our serving stack by either route. weight_mapping.py skips every mtp.* key at three separate places so the MTP weights are never loaded, and qwen36_vllm.py has no speculative path at all; on the DSpark side accept/commit is CPU-only. Nothing end-to-end can be measured, and a projected number must not be dressed up as a measured one. Deliverable: the denominator. The recorded 14.368 tok/s baseline is a single short-prompt number, which is not something you can quote an uplift against at 16k without first knowing how decode behaves with depth. Measured, goal1 profile at CTX=20480, 2x P150a TP=2, 256 tokens per rung: 26 prompt tokens -> 14.083 tok/s TTFT 0.515s 4,073 13.996 1.246 8,108 13.972 2.337 16,177 13.847 4.804 Decode is nearly FLAT in context: 1.7% decline across a 622x context increase. That follows from the architecture -- 48 of 64 layers are Gated DeltaNet with recurrent state, so only the 16 full-attention layers pay for depth. So one denominator (~14.0 tok/s) serves every rung, a speculative win would be worth as much at 16k as at 4k, and long context is not why we sit at 14 tok/s. Secondary, and counterintuitive: max_model_len costs MORE than actual depth. The same rungs on a CTX=16384 server ran ~1.4% faster at every depth purely from the smaller KV pool -- larger than the entire context effect. A baseline has to pin max_model_len, not just prompt length. Both ladders are in the record. The 26-token rung reproduces the 2026-09-04 goal-1 baseline to within 2% on a different max_model_len with the same harness, so the ladder is anchored to a reviewed number rather than free-floating. Harness fix worth naming: the first prompt calibration produced a 13,146-token prompt for a 16,384-token request, a 20% shortfall that would have mislabelled every rung. decode-bench.py now records the server's own usage.prompt_tokens plus the deviation and a within-5% flag, so a mislabelled rung shows up as a flagged record instead of a wrong column header. Every rung now lands within 1.3% of target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Owner directive: get MTP fully working and reach at least 30 tok/s on the same benchmark that currently measures ~14. This charter defines the target so it cannot be argued with -- same harness, same goal1 profile, same 4k/8k/16k rungs, byte-identical greedy output as a pass condition -- and then states two facts that constrain every plan built on it. First: MTP-1 is arithmetically disqualified. A K=1 block emits at most 2 tokens, so uplift <= 2.0 even with costless drafting and verification, which caps it at 27.99 / 27.94 / 27.69 tok/s at the three rungs. All below 30. K >= 2 is mandatory, which is the single biggest divergence from the Lottolabs reference whose device cycle is K=1 only, and it means the head runs autoregressively on its own output -- so drafter numeric error compounds, and step-K fidelity rather than step-1 fidelity sets acceptance. Second: 30 tok/s is INSIDE the target-only roofline. The runbook's measured bandwidth analysis puts the practical target-only ceiling at ~36-38 tok/s and the floor at 26-28 ms/step; we run at 71-73 ms. We are at roughly 40% of what this hardware does without any speculation. So the honest route is MTP x kernel efficiency, not MTP alone, and a programme that ignores per-token weight-read cost will likely build a correct speculative decoder that turns 14 into 21 and stops. Hence Gate 0, before any porting work: measure verify_cost_ratio on hardware for K in {1,2,3,7} at all three rungs. Packed verify is already green on the cards so this is instrumentation, not new kernel work, and it is decisive -- decode is weight-read bound, so verifying K+1 positions reads the same weights once and the ratio should sit near 1.0. If instead it lands near the third-party 1.94, then 30 tok/s needs 3.2 accepted tokens per block and no MTP-derived drafter has shown that. Predeclared: <= 1.4 at K=3 proceed, > 1.8 stop and report. The ladder after that is M1 load the mtp.* weights (weight_mapping.py skips them at three places today), M2 TP=2 head, M3 draft-token fidelity at K>1 on real hidden states, M4 verify_K, M5 the FusedCommit re-derivation for 48 v-heads and a K-way rather than dual-candidate selection -- named as the largest risk -- and M6 the benchmark itself. Recorded at precedence line 0 in CLAUDE.md; supersedes the "no MTP TP=2 head until P1 is done" line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The directive was "get MTP fully working and reach 30 tok/s". Writing the charter surfaced arithmetic that makes that framing unworkable, the owner agreed to reframe, and this is the reframed charter. GOAL-MTP-30TPS.md is renamed to GOAL-30TPS.md because it is no longer an MTP goal. tok/s = (1000 / ms_per_step) x uplift. Our step is 71.99 ms at 4k. Against the runbook's measured bandwidth analysis that decomposes as roughly 11 ms of GDN small-op launch floor (48 layers x 68 ops x ~3.4 us), 26-28 ms of weight read at peak DRAM, and ~33-35 ms of unexplained kernel inefficiency. The target-only roofline is ~36-38 tok/s, so we run at about 40% of what these cards do with no speculation at all, and Lever A alone can carry the goal. The part that changes the plan rather than just the estimates: the two levers are NOT independent. verify_cost_ratio = (t_weights + t_fixed') / (t_weights + t_fixed). Verifying K+1 positions reads the same weights once -- that sharing is the entire reason speculation can win -- but fixed cost does not amortise. When weight read dominates the ratio tends to 1.0 and speculation pays richly; when fixed cost dominates it is poor. We have ~44 ms of a 72 ms step outside the weight read, so we are in the second regime. That is the mechanical explanation of the one measured 27B result on this class of machine: 1.03-1.05x at break-even 1.94, i.e. verify plus draft costing nearly two target steps at K=1. The same runbook said MTP "is not a lever on this stack until the per-layer fixed cost falls"; this charter now explains why and orders the work by it. So Lever A (per-step cost) comes first, and it is the precondition for Lever B (MTP) being worth anything, not an alternative to it. Close half the gap and the step is 49.5 ms = 20.2 tok/s, after which 30 needs only 1.48x -- and K=1 stops being disqualified, so Lever A also changes which speculative designs are admissible. Gate 0 is now two measurements, both cheap and neither needing a speculative decoder: 0a our OWN per-step breakdown, because inheriting a third-party split is guesswork; 0b verify_cost_ratio for K in {1,2,3,7}. 0b is re-run after every Lever-A milestone -- it is the best progress indicator the programme has, since it measures whether the step has become weight-read-bound. One immediately testable A1 lead recorded: the goal2 profile enables QWEN_GDN_FUSED_DECODE, QWEN_GDN_CONV_GATES, QWEN_GDN_FUSED_INPLACE and QWEN_GDN_PACKED_QKV, and goal1 -- the baseline profile -- does not. MTP-1 stays disqualified at today's step cost (ceiling 27.69 tok/s at 16k), K>=2 remains mandatory, and step-K rather than step-1 draft fidelity sets acceptance. Section 6 now requires every result to report per-step cost and uplift separately, because a bare tok/s figure hides which lever moved and whether the denominator moved with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>agent-gateway closeout (merged): cleared
agent:*labels.Hermes kanban card (if any) is left for Hermes to archive/complete.