Qwen3.8-27B on 2x P150a: measured serving recipe (121.4 aggregate tok/s at B=8, 20.5 single-stream) #19
No reviewers
Labels
No labels
human-approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack!19
Loading…
Reference in a new issue
No description provided.
Delete branch "autoresearch/dspark-drafter-sep07"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Qwen3.8-27B on 2× P150a: a measured serving recipe
Supersedes this PR's original description entirely. That text was written 2026-09-10 around an MTP-primary plan and a 14.4 tok/s baseline. Both are obsolete: MTP was dropped by owner decision on 2026-09-12, and the baseline has moved. What follows is the current, measured state.
Headline, measured on hardware
4k context, TP=2, two cabled P150a,
vllm-tt:k2. Every row is reproduced by at least two independent harnesses; the B=8 point three times within 0.3%. Control spread on the bench is 0.09–0.48%.Read the two B=1 rows carefully — they are different recipes, and the faster one does not scale.
LM_HEAD_GATHER_MODE=noneis worth ~2 ms at B=1 and costs +31.0 ms at B=8. Anyone adopting this should take the batch-safe arm unless they genuinely serve one stream at a time.What actually produced the speedup
The single largest win was not a kernel.
process_output_decodewas reading a 32-row tile-padded logits buffer (15.9 MB) and untilizing it on the host — 10.4 ms/token of pure overhead at batch 1. Untilizing on device inside the trace (QWEN36_DECODE_LOGITS_RM=1) so the host reads a[1,1,B,vocab]row-major buffer took the step from 62.85 → 50.70 ms, output-identical. It also scales into batch gracefully (≈1 MB at B=8 instead of 15.9 MB).Four optimisation directions closed BY MEASUREMENT
Recorded so nobody re-spends the time:
QWEN36_1D_GRID_*levers fall inside a 0.48% control band. Seven decode matmuls carry TP=4 core-count constants on a TP=2 deployment (confirmed from the device's own banner, leaving most of a 110-core grid idle) — real, and worth no measurable time.gate_proj/up_projwere always bf4; the "13.61 GB all-bf8" premise in older docs is false.Known limitations, stated plainly
vllm-tt:k2is not Tracy-enabled; the profiler refuses. Getting one needs a Tracy build of the production tree._commit(mi)applies one scalar accepted index across all 48 GDN layers; the batch axis is already spent on K+1 candidates) and because the MTP tree and the performance tree are disjoint. It must not be recorded as rejected on evidence.Also in this branch
Equivalence harness for batched decode (
bench/batch-equiv-bench.py) whose load-bearing assertion is that the B per-stream hashes must be distinct — an assertion our previous harnesses would have failed, becausedecode-bench.py's counting prompt produced one identicaltext_sha256across every config and context length. Plus the batch-scaling ladder, the serving A/B driver, the low-level audit records, and incident notes (a ~10 GB pinned-memory leak over six days of uptime that OOM-killed runs and looked like a network outage).All evidence is in
bench/runs/*.jsonl; every record carries its config and caveats, and anything not timed on hardware isperformance=null,gate_complete=false.🤖 Generated with Claude Code
Result: {"status":"keep","gdn_commit_ops_shadow_k7":8,"gdn_commit_steps_shadow_commit_k7":8,"gdn_commit_copies_shadow_commit_k7":0,"gdn_commit_mb_shadow_commit_k7":0,"gdn_commit_copies_snapshot_k7":1,"gdn_commit_copies_recompute_k7":1,"gdn_commit_steps_snapshot_k7":15,"gdn_commit_steps_recompute_k7":15,"gdn_commit_state_consistent":1,"gdn_layer_bytes":1073152}Drives the production T=1 GDN decode primitive (recurrent_gated_delta_rule_decode_ttnn) through our StateTransaction protocol at TP=2. For K in {1,3,7} and every acceptance length a in [0,K], each of the three commit mechanisms (snapshot / shadow_commit / recompute) must commit the GDN recurrent state matching an autoregressive oracle replaying the identical token sequence one decode step at a time; a rejected-verify (abort) plus a later accept is also covered. Uses the 9B/27B GDN state shape [B, Nv, Dk, Dv] fp32 at B=1, Nv=8 (reduced from 32 to stay under the GDN decode kernel's compute-grid cap), Dk=Dv=128. The conv state is tracked as a per-token q shift register (the primitive covers the recurrent state, which is the D8.1 rollback core; the conv path is soft-checked). No checkpoint weights needed (deterministic per-token inputs). Registered in scripts/dspark-harness.py PROBES (name gdn-transaction -> bench/probes/dspark-gdn-transaction-tp.py).Drives the production T=1 GDN decode primitive (recurrent_gated_delta_rule_decode_ttnn) through our StateTransaction protocol on a 2-device mesh. For K in {1,3,7} and every acceptance length a in [0,K], all three commit mechanisms (snapshot / shadow_commit / recompute) commit the GDN recurrent state bit-identical to the autoregressive oracle that re-advances the pre-verify state one decode step at a time (recPCC=1.0000 for all 39 cells). A rejected verify (abort) leaves the live state exactly unchanged (torch.equal, after driving it to a non-trivial state), and a later commit after that rejection still matches the oracle advanced from the unchanged live state. The conv history is tracked as a per-token q shift register (a proxy); the primary D8.1 validation is the recurrent-state equality. performance: null (correctness only). Record: bench/runs/dspark-20260908T041702Z-6a5ffeb8.jsonl.Tenstorrent review —
autoresearch/dspark-drafter-sep07@97e818caReviewed tip 2026-09-10 ~14:37 PT (Donach's Bot “Wip”). Branch may have moved — re-check SHA before acting. Related: lanes #20 (FIR drift checklist), #21 (serve/mesh gotchas).
Executive take
Useful CPU contract + evidence discipline. Main conflict: tip/
BLACKHOLE_GUIDEmake MTP primary, while approved lane 3 is align this PR with Thatch tip techniques (precise-native draft SDPA → proposal capture → commit-only GDN) before folded T16. DSpark on silicon is not at Thatch tip parity. Keep endpoint tok/s separate from offline committed TG.What's strong
probe_passed≠gate_complete; third-party DGX/RadixArk marked)StateTransaction: snapshot / shadow_commit / recompute — right host abstraction for commit-only GDNcommit_tokens→ publish (+ rollback) — keep the regression that fails if reorderedMust-fix / decide
DSparkReference:proposals = base_logits.argmaxthen Markov only biaseslogits— do not wire those proposals into accept untilsample_block/ confidence serving sources are pinned.-tp/-tp2filename lies. Several probes are pure Torch / CPU head-index math — banner or rename; never report as mesh/TT.source.tar.gzrun bundles,generated/inspector|watcher/**— LFS/external before this is merge-reviewable.ornith/deleted butBLACKHOLE_GUIDEstill points at localornith/— fix refs to external Lottolabs only.Gap vs Thatch tip (~87 TG path)
k_chunk_size=64)SHADOW_COMMITsketch only — ≠ device tip contractShould-fix
0007; see #20)DraftAttentionPolicy,proposal_trace,commit_only_gdn) fail-closed — or explicit owner override to MTP-primary in this PR descriptionQuestions for the working agent
97e818ca…— newer Wip?sample_block/ confidence sources be pinned (SGLang rev URL)?mtp_head_parity_tp2green with a JSONL cite, or only scripted?Suggested lane-3 slice (if charter holds)
— Tenstorrent (on behalf of D. H. review ask)
Prepare reproducible DSpark development on two P150a cardsto Thatch-repro P1 on 2× P150a (override MTP-primary) — DSpark tip pathFirst hardware run of bench/probes/dspark-verify-tp.py. K in {1,3,7}, paged and unpaged, worst PCC 0.999996, VERDICT: PASS on the 2x P150a. It had never run on cards, and could not have. The probe put an implicitly- concatenated string INSIDE an f-string expression, which parses only on Python 3.12+ (PEP 701). The nix runtime shell is 3.14, so it parsed there and every prior run went through the simulator; the vllm-tt container is Python 3.10, where it was a SyntaxError before the first line executed. That single incompatibility is why this probe's whole recorded history is simulator runs. Worse, the record it writes hard-coded "chips": "P150a x2 (simulator)". Running it on hardware would have produced a false record claiming the opposite of what happened. The label is now derived from TT_METAL_SIMULATOR, which is what actually selects the backend, so neither direction can be mislabelled; the timing-claim caveat is likewise conditioned on the real runtime. What this is NOT: it is target-only packed-SDPA kernel parity -- the packed K+1 call reproducing K+1 sequential single-position calls. There is no argmax, no accept decision, no GDN state and no commit anywhere in it. It does not close the shared-verifier row, and the record says so; gate_complete=false, performance=null. Next: the single decision owner (select_prefix -> Decision), folding in the EOS semantics where the tip emits no correction token but we currently would. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>P1 progress — 5 commits (
4922f17..ba3e7ff)Tip re-verified at
b2f9ffe(ci/qwen-hardware-correctness). I enumerated every remote branch and PR ref: that branch is the most recently updated, 6 days ahead ofmainand of everything else, and is the head of their PR #7. None of our pinned P1 files changed upstream.Row status
Hardware results
1. The tip's FP32-intermediate C++ patch is reproduced on our 2× P150a. The checklist said it couldn't be built here; that was stale. Image
vllm-tt:k2-fp32exists, and the gate is observed engaging in stderr, not inferred:18-run sweep (3 seeds × 3 shapes × 2 arms, HiFi4): patched ≥ baseline in 9/9 pairs, gate engaged on all 9 patched and zero baseline runs — consistent, not seed noise. But the benefit is strongly context-dependent: mean PCC-error reduction 39.0% at CTX=2048 (n=6) vs 8.8% at CTX=4096 (n=3), and absolute error roughly doubles with context on both arms. A serving drafter runs at long context, so this weakens the case that the patch is where the tip's ~87 TG comes from.
Separately, at LoFi the unpatched op fails the all-ones sanity outright (frac_near_1 = 0.0) and the patch restores it to 1.0000 — but both LoFi arms miss PCC_MIN=0.999, so LoFi isn't a usable drafter fidelity either way.
2. Packed verify is green on the real cards, for the first time. K ∈ {1,3,7}, paged and unpaged, worst PCC 0.999996.
Three pre-existing breakages, found by running things
0118971.tests/test_dspark_reference.py:88hadp.clone(, allow_argmax_placeholder=True)— anast.parsefailure, so pytest collection died for the entiretests/directory, not just that file.dspark-verify-tp.pycould never run in the container. It put an implicitly-concatenated string inside an f-string expression — valid only on Python 3.12+ (PEP 701). The nix shell is 3.14 so it parsed there; the container is 3.10, where it was aSyntaxError. That is why its entire recorded history is simulator runs. It also hard-coded"chips": "P150a x2 (simulator)"into its own record, so a hardware run would have produced a false record. Now derived fromTT_METAL_SIMULATOR._conv_pccwas fail-open twice over. It skipped any conv tap that wasNoneon either side and returned 1.0 when nothing remained. A tap filled on one side and empty on the other is a real fill-pattern divergence — exactly the off-by-one a commit-only port introduces — and it was dropped rather than compared.Fail-open holes closed on the serving path
e2e.pyassignedtransaction.live = pre_livedirectly on the publication-failure path — a rollback engine outside the transaction, the clearest violation of "keep single verifier owner". NowStateTransaction.rollback(), which fails closed if called mid-verify.forced_acceptance_lengthsat unguarded on the accept path. It bypasses the target comparison entirely, so a cycle could "accept" tokens the verifier never agreed with. Every test passed an empty iterator, so nothing used it — pure fail-open surface. Refused now without explicit opt-in.Corrections to this branch's own docs
hybrid_draft.py/lookup_draft.py/greedy_verify.py/greedy_session.py/lookup_acceptance.py. Those are not the capture path — they're a host-side n-gram coordinator that never touches a tensor; their own README says "a host coordinator, not a TT device executor or a throughput result". The real path isscripts/ci/dspark_markov.py, ~10 lines.optimisation/sim/gdn-commit.pyis not the commit-only contract — it's a synthetic publication fixture, andtest_commit_completion.pyis an AST meta-test overgdn-multitoken.py. Semantics live ingdn_device_loop_state.py.dspark-verify-tp.pyas the verifier. It's a kernel-parity probe — no argmax, no accept, no GDN, no state.Do not chase T32
Their newest line (31 draft queries / 32 rows) has zero hardware evidence — two CPU-simulator correctness runs plus host unit tests. Their own words: "they do not mount the cards or measure hardware speed", "Not run; device-state and combined PP/CTX/TG admission still required." Unpromoted additional arm; reuses the precise-native draft SDPA unchanged. Their only real-hardware gain in that range is T16 gate/up fusion at +1.4–2.0% TG, which they say must not be presented as resolving 200 TG. Banked-proposal and native-slot-GDN arms are both recorded by them as not promoted, with no meaningful speedup.
What is NOT claimed
Every record is
gate_complete=false,performance=null. Rows 2–3 are CPU semantics only. The commit-only mechanism cannot be validated without hardware: single-launch DMA atomicity, native+checkpoint from one staged buffer, the 32-byte two-face NOC scatter and its tiled offset arithmetic, inactive-slot non-interference, per-chip mesh consistency. No TG number here is ours, and none is implied.Open provenance gap, recorded in the run file: the overlay Dockerfile asserts a
b9bb5825tt-metal build base while our records pin k2's at1227e182, and neither.soexposes a version string to settle it.Suite: 121 passed, 8 failed — the 8 being the pre-existing
pyexpatImportErrorgap.Next
Single decision owner:
select_prefix→Decision(emitted, accepted, state_rows, next_input, finished), with an invariant thatcommit_tokensrefuses a token list it did not receive a matchingDecisionfor. Folding in the EOS divergence — the tip emits no correction token when EOS lands inside the accepted prefix; we currently would.🤖 Generated with Claude Code
MTP unparked (owner direction) — first hardware result: the 27B MTP head's draft token.
Commit
8b8a818. New probebench/probes/mtp_draft_token_fidelity.py; evidencebench/runs/mtp-draft-token-20260911T144311Z.jsonl(gate_complete=false,performance=null).Why a new probe.
mtp_head_parity.pycompares hidden-state taps (cosine/Pearson) and stops atfinal_norm, feeding the predictor layer an oracle-sourcedfc_out— so the pre-fc fusion never ran on TT. But a drafter emits a token, and a 0.985-cosinemlp_outsays nothing about an argmax over 248,320 logits. Nothing in the repo had asked that question.This runs the whole head on one P150a — embedding, both pre-fc RMSNorms, the [5120,10240]
mtp.fcmatmul, the gated full-attention layer, final norm, shared LM head. No tap is oracle-sourced any more, which also closes the boundary the worklog flagged.Measured, 128 hidden states:
So the TT dtype chain flips roughly one draft in six. That matters because MTP's margin is thin — the third-party 27B figure is acceptance 2.04 against break-even 1.94.
What this does not establish. The hidden states are seeded Gaussian proxies. They give abnormally flat logit landscapes — exactly the regime where a small numeric perturbation flips an argmax. 82.8% is most likely a pessimistic lower bound. The next action is re-running against real captured 27B target hidden states; that substitution decides whether MTP is worth a TP=2 port.
The probe voided itself once before producing this number. At n=16 a single 5% perturbation of
finalraised agreement 11→12 — a few-percent nudge is below the TT dtype error itself — so the run refused its own 68.8% headline. Replaced with a monotone sweep (25/18/7/0 at 0.05/0.25/1.0/4.0 relative RMS), a mismatched-oracle control (7/128 vs 106/128 paired), and a float32 anchor against the committed fixture (token 9767, seed 42 → argmax 79).Two host traps recorded in
CLAUDE.md, both paid for here:embed_tokensandlm_headare BF16[248320,5120]; upcasting both to f32 (asmtp_head_oracle.load_weightsdoes) is a silent exit 137.In parallel, an
agyagent is measuring the DSpark drafter's acceptance rate against real 27B target logits in a sibling worktree, on the same predeclared criterion (mean accepted length ≥3.0 at K=7 justifies the GDN device-DMA port; below ~2.0 does not).The owner asked for true uplift at 4k/8k/16k over the ~15 tok/s no-drafter baseline. Half of that is deliverable today and half is not, so this commit delivers the half that is and says clearly why the other half isn't. Not deliverable: there is no speculative decoding in our serving stack by either route. weight_mapping.py skips every mtp.* key at three separate places so the MTP weights are never loaded, and qwen36_vllm.py has no speculative path at all; on the DSpark side accept/commit is CPU-only. Nothing end-to-end can be measured, and a projected number must not be dressed up as a measured one. Deliverable: the denominator. The recorded 14.368 tok/s baseline is a single short-prompt number, which is not something you can quote an uplift against at 16k without first knowing how decode behaves with depth. Measured, goal1 profile at CTX=20480, 2x P150a TP=2, 256 tokens per rung: 26 prompt tokens -> 14.083 tok/s TTFT 0.515s 4,073 13.996 1.246 8,108 13.972 2.337 16,177 13.847 4.804 Decode is nearly FLAT in context: 1.7% decline across a 622x context increase. That follows from the architecture -- 48 of 64 layers are Gated DeltaNet with recurrent state, so only the 16 full-attention layers pay for depth. So one denominator (~14.0 tok/s) serves every rung, a speculative win would be worth as much at 16k as at 4k, and long context is not why we sit at 14 tok/s. Secondary, and counterintuitive: max_model_len costs MORE than actual depth. The same rungs on a CTX=16384 server ran ~1.4% faster at every depth purely from the smaller KV pool -- larger than the entire context effect. A baseline has to pin max_model_len, not just prompt length. Both ladders are in the record. The 26-token rung reproduces the 2026-09-04 goal-1 baseline to within 2% on a different max_model_len with the same harness, so the ladder is anchored to a reviewed number rather than free-floating. Harness fix worth naming: the first prompt calibration produced a 13,146-token prompt for a 16,384-token request, a 20% shortfall that would have mislabelled every rung. decode-bench.py now records the server's own usage.prompt_tokens plus the deviation and a within-5% flag, so a mislabelled rung shows up as a flagged record instead of a wrong column header. Every rung now lands within 1.3% of target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Baseline context ladder measured — the uplift denominator. And the reason there is no uplift number yet.
Commits
c70e86b(harness) and0b90043(measurement). Evidence:bench/runs/baseline-context-ladder-20260911T150910Z.jsonl.The blocker, stated plainly
There is no speculative decoding in our serving stack, by either route. In
vllm-tt:k2,models/demos/blackhole/qwen36/tt/weight_mapping.pyskips everymtp.*key at three separate places (50, 135, 425) — the MTP head's weights are never loaded — andqwen36_vllm.pyhas no speculative path at all. On the DSpark side accept/commit is CPU-only and the device DMA commit is unported. So no end-to-end uplift is measurable today, and I am not going to present a projection as one.What is measurable is the two halves that multiply into it. This is the first.
Measured — target-only decode vs context
2× P150a, mesh
P300 (1,2)TP=2,vllm-tt:k2, goal1 profile atCTX=20480 SEQS=1, 256 tokens streamed per rung, greedy, single stream:Decode is nearly flat in context — 1.7% decline across a 622× context increase. 48 of the 64 layers are Gated DeltaNet carrying recurrent state rather than a growing KV cache, so only the 16 full-attention layers pay for depth. TTFT scales as expected; it is prefill and excluded from the decode rate.
Consequences: one denominator (~14.0 tok/s) serves every rung; a speculative win would be worth as much at 16k as at 4k; and long context is not why we sit at 14 tok/s — per-token weight reading is.
Secondary, counterintuitive:
max_model_lencosts more than actual depth. The same rungs on aCTX=16384server ran ~1.4% faster at every depth, purely from the smaller KV pool — larger than the entire context effect. A baseline has to pinmax_model_len, not just prompt length. Both ladders are in the record.Tie-back: the 26-token rung reproduces the 2026-09-04 goal-1 baseline (14.368 / 14.418) to within 2%, same harness, different
max_model_len.Harness
bench/decode-bench.py --prompt-tokens Nbuilds a deterministic prompt and records the server's ownusage.prompt_tokens. The first calibration silently produced a 13,146-token prompt for a 16,384-token request — a 20% shortfall that would have mislabelled every rung. It now also records the deviation and a within-5% flag, so a mislabelled rung is a flagged record rather than a wrong column header. All rungs now land within 1.3%.scripts/vllm-tt-serve.sh goal1honoursCTX/SEQSand changes nothing else. This matters:goal2/goal3also enable bf8 KV and a bf4down_proj, so a number measured there is not a valid denominator.bench/speculative-uplift.pycomposesuplift = (mean_accepted + 1) / (verify_cost + draft_cost), labels everythingPROJECTION, and prints the costless ceiling and break-even. It refuses to project unless it first reproduces the only end-to-end 27B figure anyone has measured — the third-party 1.03–1.05× at acceptance 2.04 / break-even 1.94 — from their own inputs. It returns 1.052×.That anchor is sobering: their break-even of 1.94 means verify + draft cost nearly two target steps at K=1, which is why 2.04 accepted tokens bought only 3–5%.
Next
Measure
verify_cost_ratioon hardware — time one packed verify of K+1 positions against one target decode step at these same depths. Packed verify is already green on the cards, so this is a small probe, and it is the last unknown in the formula that does not require a speculative decoder to exist. Theagyagent has been redirected to report DSpark acceptance per rung at 4k/8k/16k so its numerator composes with this denominator.Owner directive: get MTP fully working and reach at least 30 tok/s on the same benchmark that currently measures ~14. This charter defines the target so it cannot be argued with -- same harness, same goal1 profile, same 4k/8k/16k rungs, byte-identical greedy output as a pass condition -- and then states two facts that constrain every plan built on it. First: MTP-1 is arithmetically disqualified. A K=1 block emits at most 2 tokens, so uplift <= 2.0 even with costless drafting and verification, which caps it at 27.99 / 27.94 / 27.69 tok/s at the three rungs. All below 30. K >= 2 is mandatory, which is the single biggest divergence from the Lottolabs reference whose device cycle is K=1 only, and it means the head runs autoregressively on its own output -- so drafter numeric error compounds, and step-K fidelity rather than step-1 fidelity sets acceptance. Second: 30 tok/s is INSIDE the target-only roofline. The runbook's measured bandwidth analysis puts the practical target-only ceiling at ~36-38 tok/s and the floor at 26-28 ms/step; we run at 71-73 ms. We are at roughly 40% of what this hardware does without any speculation. So the honest route is MTP x kernel efficiency, not MTP alone, and a programme that ignores per-token weight-read cost will likely build a correct speculative decoder that turns 14 into 21 and stops. Hence Gate 0, before any porting work: measure verify_cost_ratio on hardware for K in {1,2,3,7} at all three rungs. Packed verify is already green on the cards so this is instrumentation, not new kernel work, and it is decisive -- decode is weight-read bound, so verifying K+1 positions reads the same weights once and the ratio should sit near 1.0. If instead it lands near the third-party 1.94, then 30 tok/s needs 3.2 accepted tokens per block and no MTP-derived drafter has shown that. Predeclared: <= 1.4 at K=3 proceed, > 1.8 stop and report. The ladder after that is M1 load the mtp.* weights (weight_mapping.py skips them at three places today), M2 TP=2 head, M3 draft-token fidelity at K>1 on real hidden states, M4 verify_K, M5 the FusedCommit re-derivation for 48 v-heads and a K-way rather than dual-candidate selection -- named as the largest risk -- and M6 the benchmark itself. Recorded at precedence line 0 in CLAUDE.md; supersedes the "no MTP TP=2 head until P1 is done" line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The directive was "get MTP fully working and reach 30 tok/s". Writing the charter surfaced arithmetic that makes that framing unworkable, the owner agreed to reframe, and this is the reframed charter. GOAL-MTP-30TPS.md is renamed to GOAL-30TPS.md because it is no longer an MTP goal. tok/s = (1000 / ms_per_step) x uplift. Our step is 71.99 ms at 4k. Against the runbook's measured bandwidth analysis that decomposes as roughly 11 ms of GDN small-op launch floor (48 layers x 68 ops x ~3.4 us), 26-28 ms of weight read at peak DRAM, and ~33-35 ms of unexplained kernel inefficiency. The target-only roofline is ~36-38 tok/s, so we run at about 40% of what these cards do with no speculation at all, and Lever A alone can carry the goal. The part that changes the plan rather than just the estimates: the two levers are NOT independent. verify_cost_ratio = (t_weights + t_fixed') / (t_weights + t_fixed). Verifying K+1 positions reads the same weights once -- that sharing is the entire reason speculation can win -- but fixed cost does not amortise. When weight read dominates the ratio tends to 1.0 and speculation pays richly; when fixed cost dominates it is poor. We have ~44 ms of a 72 ms step outside the weight read, so we are in the second regime. That is the mechanical explanation of the one measured 27B result on this class of machine: 1.03-1.05x at break-even 1.94, i.e. verify plus draft costing nearly two target steps at K=1. The same runbook said MTP "is not a lever on this stack until the per-layer fixed cost falls"; this charter now explains why and orders the work by it. So Lever A (per-step cost) comes first, and it is the precondition for Lever B (MTP) being worth anything, not an alternative to it. Close half the gap and the step is 49.5 ms = 20.2 tok/s, after which 30 needs only 1.48x -- and K=1 stops being disqualified, so Lever A also changes which speculative designs are admissible. Gate 0 is now two measurements, both cheap and neither needing a speculative decoder: 0a our OWN per-step breakdown, because inheriting a third-party split is guesswork; 0b verify_cost_ratio for K in {1,2,3,7}. 0b is re-run after every Lever-A milestone -- it is the best progress indicator the programme has, since it measures whether the step has become weight-read-bound. One immediately testable A1 lead recorded: the goal2 profile enables QWEN_GDN_FUSED_DECODE, QWEN_GDN_CONV_GATES, QWEN_GDN_FUSED_INPLACE and QWEN_GDN_PACKED_QKV, and goal1 -- the baseline profile -- does not. MTP-1 stays disqualified at today's step cost (ceiling 27.69 tok/s at 16k), K>=2 remains mandatory, and step-K rather than step-1 draft fidelity sets acceptance. Section 6 now requires every result to report per-step cost and uplift separately, because a bare tok/s figure hides which lever moved and whether the denominator moved with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>5ccce9c79785b9065737Goal reframed: ≥ 30 tok/s is a two-lever goal, and MTP is the second lever.
docs/GOAL-30TPS.md, commit5ccce9c, installed at precedence line 0 inCLAUDE.md.The directive was "get MTP fully working and reach 30 tok/s". Writing the charter surfaced arithmetic that makes that framing unworkable.
The decomposition
tok/s = (1000 / ms_per_step) × uplift. Our step is 71.99 ms at 4k. Against the runbook's measured bandwidth analysis:The target-only roofline is ~36–38 tok/s. We run at roughly 40% of what these cards do with no speculation at all, so Lever A alone can carry the goal.
The part that changes the plan, not just the estimates
The two levers are not independent:
Verifying K+1 positions reads the same weights once — that sharing is the entire reason speculation can win. Fixed cost does not amortise. When the weight read dominates, the ratio tends to 1.0 and speculation pays richly; when fixed cost dominates, it is poor. We have ~44 ms of a 72 ms step outside the weight read.
That is the mechanical explanation of the one measured 27B result on this class of machine: 1.03–1.05× at break-even 1.94 — verify plus draft costing nearly two target steps at K=1 is exactly the signature of a fixed-cost-dominated step. The runbook already said MTP "is not a lever on this stack until the per-layer fixed cost falls"; the charter now explains why and orders the work by it.
So Lever A comes first, and it is the precondition for Lever B being worth anything — not an alternative to it.
Lever A also changes which speculative designs are admissible: MTP-1 is disqualified at today's step cost (ceiling 27.69 tok/s at 16k, so K ≥ 2 is mandatory), but at a 49.5 ms step K=1's ceiling is 40 tok/s.
Gate 0 — two cheap measurements, before any porting
verify_cost_ratiofor K ∈ {1,2,3,7} at all three rungs. Packed verify is already green on the cards, so this is instrumentation. Predeclared: ≤ 1.4 at K=3 → build Lever B now; > 1.8 → do not port a commit kernel, do Lever A and re-measure.0b is re-run after every Lever-A milestone. It is the programme's best progress indicator, because it measures whether the step has become weight-read-bound.
One immediately testable A1 lead: the
goal2profile enablesQWEN_GDN_FUSED_DECODE,QWEN_GDN_CONV_GATES,QWEN_GDN_FUSED_INPLACEandQWEN_GDN_PACKED_QKV, andgoal1— the baseline profile — does not.§6 now requires every result to report per-step cost and uplift separately: a bare tok/s figure hides which lever moved and whether the denominator moved with it.
Gate 0a part 2 — DRAM bandwidth measured; the target-only path is closed
bench/dram-saturation.pyhad never been run.SATURATION-AND-SRAM.md:387called it the unknown that bounds everything else. Now measured on the card, model not loaded, device exclusive —bench/runs/gate0a-dram-saturation-20260911T183617Z.jsonl.430 GB/s does not exist.
Three findings, all unwelcome, which is why they matter:
The goal1 target-only ceiling is 27.81 tok/s at peak and 24.32 on decode's own grid — below the 30 tok/s target with zero fixed cost and a perfect kernel. Even all-MLP-bf4, which we don't have and which costs accuracy, reaches 34.89 / 30.51.
So speculation is no longer the optional half of the charter — it is load-bearing, with nothing behind it. Lever A stays the first move only because speculation cannot pay until the step is weight-read-bound.
This charter has now been corrected twice in one day by its own gates, both downward, both because an inherited third-party figure did not survive measurement. Commit
5b00938.Still running: A1 GDN-fusion A/B (holds the lock now), and the DSpark acceptance-rate sweep whose live log is showing accepted=0 on 14 of its first 15 blocks at 4k/K=15.
A1 (GDN fusion) — PASS, and it was a configuration gap
Measured A/B on the cards, same image and profile, only env differs —
bench/runs/a1-gdn-fusion-ab-20260911T184615Z.jsonl.QWEN_GDN_FUSED_DECODE/CONV_GATES/FUSED_INPLACE/PACKED_QKValready exist and are enabled in the goal2 profile. goal1 — the profile every baseline number in this repo was measured in — does not set them. So this is a gap we had been paying for rather than new work, and the gain grows with context. The base arm reproduces the recorded baseline (14.13/14.00/13.86 vs 13.85–14.00), which anchors the comparison.One caveat I want on the record because it will bite us later. The benchmark generates "count from 1 to 300", so both arms and all three rungs produce an identical
text_sha256(bafed7a9…). The byte-identity check therefore proves only that fusion did not break trivial counting — not numeric equivalence on hard tokens. Worse: that same degeneracy would hand speculative decoding an unrepresentatively high acceptance rate. MTP uplift must never be measured on this generation alone, and we need a prompt-dependent output arm before any Lever B number is believable.Where Lever A now stands: 65.54 ms against a 35.95 ms measured-peak weight-read floor. A1 closed roughly a seventh of the gap; ~30 ms of non-weight-read cost remains. It does not change the Gate 0a part 2 conclusion. Commit
47f2b71.The plan changed: Tenstorrent already built our Lever B
A scout agent reported this; a scout claim is not evidence, so I checked every structural claim against the GitHub API and the raw sources before writing it down. Record:
bench/runs/scout-pr55548-verification-20260911T192500Z.jsonl.tenstorrent/tt-metalPR #55548 — "[Feat]: MTP Spec Decode for Qwen3.8-27B for QB2", branchatupe/qwen36-mtp-v2, head96f3f04102028cd85c23c2162914d5a0e7b37cc6, open and unmerged, 53 files, +8294, Apache-2.0, by a Tenstorrent engineer.Verified present, not inferred:
tt/spec_decode.py(+831),tt/mtp.py(+248),tt/gdn/tp.py(+753), theweight_mapping.pychange that unblocks exactly themtp.*keys we found skipped at lines ~50/135/425, a new C++fused_recurrent_gated_delta_rulettnn op with compute/reader/writer kernels, and aspec_multi_posmode insdpa_decode.The decisive line,
spec_decode.py:53:It is TP-only. Lottolabs' MTP raised
NotImplementedErrorfornum_devices != 1— the opposite constraint, and the reason we wrote that path off. Ranks 1 (K>1) and 2 (TP≥2) of the brief are satisfied by one artifact, inside our own model's repo tree.What does not transfer. The author's 50.45 tok/s at 4k and 2.62× at ISL 128 are on QuietBox 2 with four dies, where each card streams ~6.8 GB/token against our 13.61 at TP=2. Their number sits near their roofline, which is about twice ours. The mechanism transfers; the tok/s does not, and I won't quote it as a target.
Two real costs. It needs tt-metal built from an unmerged branch — that collides with "pins move as a set" and is a packaging problem, not a modelling one. And it is lossless only conditionally: the PR says it matches greedy where plain decode is confident (top-2 gap ≥ 2.0) and that near-ties "can differ and are logged". Our charter makes byte-identity a pass condition. That is a different standard, and we should settle it deliberately rather than discover it in a benchmark.
A1b — Lever A configuration work is nearly exhausted
bench/runs/a1b-flag-sweep-20260911T193122Z.jsonl. Re-running the identicalfusedarm in a fresh session differed by 0.33%, which finally bounds noise on this bench.fused)fused(A1)+ BATCHED_GROUPED=0+ GDN_DECODE_BF16 + STATE_BF16+ bothGDN bf16 is +2.7–3.0%, ~9× the noise floor, real.
BATCHED_GROUPED=0is noise-order and not established; the two don't compose.That 3% is not yet free — those flags hold the recurrent state in bf16, i.e. they change numerics, and our sha check is degenerate (counting survives almost any perturbation). Gated on a real equivalence arm.
Lever A: 71.99 → 63.27 ms, 1.126×, entirely from flags that already existed and
goal1never set. The remaining ~27 ms above the 35.95 ms floor is kernel work — which is precisely what #55548 contains.Commits
f96c7a8,b544bc7.The owner cleared taking any pin or branch and applying patches freely, so this takes the shortest honest route to a measurement. We did not compile tt-metal. cfx-llm2 has 4 cores and ~4 GB free RAM once hugepages are reserved, so a from-source build there is not realistic. The branch's own CI already publishes a matching wheel -- cp310 / manylinux_2_35 against the serving image's Ubuntu 22.04, Python 3.10.19, glibc 2.35 -- so the image is assembled from that. Pinned as a set at 96f3f04102028cd85c23c2162914d5a0e7b37cc6: both the ttnn wheel and the qwen36 model tree. The branch head moved to f80137ca while we were cloning it, which is exactly the mismatched-pair trap our own rules warn about, so the tree is checked out at the wheel's sha rather than at the branch tip. Verified in the built image: ttnn.transformer.fused_recurrent_gated_- delta_rule is present and sdpa_decode carries the spec argument. One patch was required. text_demo.py maps MESH_DEVICE to a mesh shape and has no entry for P300, so our 2-die rig silently fell back to their 4-die (1,4). Added P300 -> (1,2). Two things recorded because they change the plan rather than decorate it: 1. This image DOWNGRADES ttnn 0.79.0.dev+cfx -> 0.75.0rc10. The vllm-tt plugin is built against 0.79 and is not expected to work here. vllm-tt:k2 is untouched and remains the serving image. 2. tt/qwen36_vllm.py in this PR has NO speculative-decode wiring. MTP lives in spec_decode.py driven by text_demo.py. Measuring the uplift is a wheel swap; shipping it through our server is a real integration. The scout's "cherry-pick and adapt" framing understated this, and it is better to know now than after committing to a date. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Pushed: everything through the local tt-metal build (
370bb7b)Correction I owe the record: the pin was never "older"
1227e18296f3f04160053f1fThe bump moved us a month forward. Three version strings disagree with recency and all three mislead:
0.77.0-dev20260813is hand-written in our own flake.nix and stale;0.75.0rc10is the PR's CI release-branch numbering; and the container's0.79.0.dev20260903+cfxis a cfx fork dated 09-03 — older than the PR despite the larger number. I've annotated the flake string so nobody repeats the confusion. Compare commit dates, never version numbers.But we now build main + the PR, not the PR alone
The PR branch is 80 commits behind main. The merge onto current main is conflict-free (53 files), so we take both rather than forfeiting upstream work — and it leaves room to stack further open PRs. SFPI followed the source to 7.76.0 automatically, which is the self-consistency the hand-assembled image never had.
Why the CI-wheel shortcut failed (hours, and a deceptive failure mode)
Assembling an image mixes four independently versioned things — the wheel's headers, its
.so, the image's SFPI, and the image'sTT_METAL_HOMEtree — and each mismatch only surfaces after a ~25 minute weight load:models/experimental/gated_attention_gated_deltanet(the PR patches it too);brisccompile failure deep intensor_accessor_args.h, never as a version error;TT_METAL_HOMEon the 0.79 tree while headers came from the 0.75 wheel;static assertion failed: Index out of range … (3130 < 7).Written up in
docs/MTP-BUILD-WORKFLOW.md. Build the tree; do not assemble one.Two self-inflicted traps now in CLAUDE.md
docker stop-ing a live tt-metal container wedges the cards. Next device open fails "failed to initialize FW! Try resetting the board" — and it presents as a version problem. Our known-good 0.79 image failed identically, which is what disproved it.sudo tt-smi -rfixes it (a reset, not a flash).tt-smi -rreturns before the boards are ready; launching immediately givesRead 0xffffffff over PCIe. Verify with a mesh-open probe first.Measurements landed
~26 ms of the 63 ms step is still unexplained, and that is where the remaining headroom lives.
The previous section read the first ruling as "per-user latency" => "batching is excluded". That was an over-reading and is retracted here. Production will run B in [4,8]; the figure of merit is per-user tok/s AT that batch, with aggregate a secondary benefit. A B=32 point at 8.6 tok/s per user is a worse answer than B=4 at 18, and must not be proposed. Consequences recorded: - Baseline restated to the measured production step (50.70 ms / 19.7 tok/s at 4k B=1) rather than the retired 63.27 ms figure. - The step's dominant terms (weight read, per-op dispatch) are flat in B, so batching should be cheap in latency here -- but step(B) has NEVER been measured on the production arm, and the P0 itemisation cannot predict it because P0 profiled the composed GDN path, not the served model. step(B) for B in {1,2,4,8} is now a required measurement. - Lever A is worth MORE: it cuts flat terms, so the saving is enjoyed once by every one of the B users -- the only lever improving per-user and aggregate together. - Lever B (MTP) gets harder at B>1. Sequences accept different draft lengths, so a batched verify must either commit the batch minimum (discarding accepted tokens) or support ragged commit; which of those spec_decode.py does is unknown. A B=1 MTP result is necessary but no longer sufficient. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>We believed two things that a careful read of the tree disproves. First, that the open question at B>1 was WHICH commit policy speculative decode uses -- batch minimum, or ragged. It uses neither, and there is no assert to trip either. `generate()` takes one flat prompt id list, `p` is a scalar int, and `_commit(mi)` applies ONE accepted-prefix index to all 48 GDN layers. The batch axis is already spent: the K+1 CANDIDATES of a single sequence are what `use_fullbatch_verify` batches over, and attention's `_SPEC_SDPA_L1_FIT` admits T in {4,8,12} = K+1, never B. So a B=1 acceptance rate cannot be carried to the owner's B=4-8 operating point by a flag; it needs per-sequence positions, a [B,K+1] verify and ragged commit, which is a redesign. Second, that MTP could be measured on the validated production configuration. It cannot, with the images on this host. vllm-tt:k2 -- the image behind the 50.70 ms/token result -- has no mtp.py, no spec_decode.py, and zero of the six speculation hooks across model.py, gdn/tp.py and attention/tp.py. vllm-tt:mtp has the complete spec stack and none of k2's conv_gates / packed_qkv / fused_inplace GDN work; QWEN36_DECODE_LOGITS_RM is in neither image (it was bind-mounted). The branches are disjoint, so an MTP run yields a valid internal ratio and an absolute ms/token that is NOT the production arm's. Also fixes the harness that failed its first attempt: both arms now go through the production dispatch toggled by QWEN36_SPEC (not QWEN36_MTP, which only gates LOADING the head) with engagement asserted in both directions from the demo's own log lines; exception text is truncated (a JIT build failure wrote a 663 MB JSONL for two records); a failed arm exits hard rather than deadlocking in teardown while holding the cards; and the flag set drops two knobs this tree does not implement. Adds smoke-demo.sh, a zero-custom-code image-health gate to settle whether the kernel-build failure indicts the image or the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>that has none of our performance work Two structural facts, both verified independently of the agent that found them, and together they reshape the plan. 1. The MTP tree and the validated-performance tree are DISJOINT. vllm-tt:k2 has QWEN_GDN_CONV_GATES and QWEN_GDN_PACKED_QKV but no spec_decode.py, no mtp.py and zero hits for capture_verify_trace. vllm-tt:mtp is the exact mirror. QWEN36_DECODE_LOGITS_RM is in NEITHER -- it was bind-mounted. So a SPEC=0 control on the only tree that has MTP cannot land at 50.70 ms/token and will land nearer 63 for a known reason. An A/B there gives a sound internal RATIO; that ratio must not be multiplied onto 50.70 ms. Merging the branches is an integration task now on the critical path. 2. Speculation has no batch dimension over sequences at all -- it is neither batch-minimum nor ragged commit, and there is no assert to trip because nothing ever offers it a batch. _commit(mi) applies one SCALAR accepted index across all 48 GDN layers, so a per-sequence accepted length is not expressible. The batch axis is already spent on CANDIDATES: use_fullbatch_verify batches the K+1 candidates of one sequence and _SPEC_SDPA_L1_FIT admits T in {4,8,12} = K+1, never B. At B>1 it fails on shape, not silently. The owner's operating point is batch 4-8 with per-user latency decisive. A B=1 acceptance number therefore does NOT transfer to it by setting a flag: batched speculation needs per-sequence positions, a [B,K+1] verify and ragged commit. That is a redesign, not a configuration. Also corrects the previous record: the failed run's control arms were NOT secretly speculative (the driver set QWEN36_SPEC=0 and called the spec entry point directly for treatment). And prose_stock, the demo's own "4k" arm, is 2642 tokens of one tiled paragraph at 0.04 distinct-token fraction -- unusable for acceptance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Observed read-only from this session while waiting for the lock. dmesg has two global OOM kills, the second naming VLLM::EngineCor directly (total-vm 77.8 GB, anon-rss 2.46 GB) at 00:07:36Z. free shows 15 GB total, 13 used; tt-metal's ~10 GB of hugepages leaves the ~5.5 GB CLAUDE.md already documents. That is what the two "server never came up" arm logs are. Their visible tail is only the downstream "Engine core initialization failed ... Failed core proc(s): {}" -- and that EMPTY set is the signature of a SIGKILLed engine core, not of an engine-core exception, which is why the logs carry no root cause of their own. Neither log contains a card-wedge signature. Worth being ready for: the OOM was a SIGKILL of a tt-metal process that was at least loading weights, which is the situation CLAUDE.md says can leave the next device open failing "Timeout waiting for physical cores to finish" / "failed to initialize FW!" and looking like a version mismatch. This session reset nothing and opened no device. The arm log also happens to carry the model's own resolved-grid banner, which turns two source-only audit findings into hardware-log evidence: the LM head really does run on ttnn-auto with no program config, and the hardcoded decode grids reach the device as 44/44/33/44/33/33 on a 110-core chip. The record includes the per_core_N each resolves to and one arithmetic worked example, and claims no time saving from any of it -- a 2.35x standalone matmul win has already measured 0-2% SLOWER in the model once, so the sweep has to say. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>/tmp survived the reboot, so the first attempt's logs were readable after all, and two things in the record needed fixing. The lock was NOT stranded. I had flagged it as probably stranded on the assumption that a SIGKILL skipped my EXIT trap. In fact only the containerised EngineCore was killed; the runner survived, reported "server never came up" and logged "lock released". Being detached is why that log existed to read at all. The OOM diagnosis, which this record originally relayed second-hand, is now corroborated independently and precisely: the core's last line is "Warm state_dict: 851 weights, 80 real host weights from the sidecar", after which it died with no traceback of its own and the API server reported "Failed core proc(s): {}". An empty failed-proc set with no core traceback is death by signal, landing exactly on the step that allocates host weights. Still zero points, still no baseline, still is_reference=false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Thatch-repro P1 on 2× P150a (override MTP-primary) — DSpark tip pathto Qwen3.8-27B on 2x P150a: measured serving recipe (121.4 aggregate tok/s at B=8, 20.5 single-stream)