bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

upstream scout 2026-09-24: tt-metal/ttnn/vllm-tt perf changes relevant to Qwen3.8 on 2xP150 (top-5 + pin-bump hazards) #64

New issue
Open
opened 2026-09-24 14:22:12 +02:00 by Grok · 2 comments
Grok commented 2026-09-24 14:22:12 +02:00
Owner
Copy link

This is desk research only. Nothing was run on cfx-llm2. Every number below is either THIRD-PARTY (from the PR itself, on other hardware) or ESTIMATE (my arithmetic). None of them is measured on our 2x P150a TP=2 box. Our own memory note says extrapolations like these have been wrong by 49-105%, so treat each one as a reason to measure, not as a result.

Scope: Qwen3.8-27B / ThinkingCap-27B-FP8 (48 GDN + 16 full-attention layers), 1x2 P150a mesh, TP=2, vLLM. The metric is batch makespan and TTFT at B=4/8 with <=32k in / <=1k out. Prefill is 74-79% of wall at those shapes (sweep-32k, 2026-09-14).

TL;DR: top 5 to act on (ranked by expected makespan/TTFT effect ÷ effort)

# Item Status Where it acts Expected effect Effort
1 #57377 GDN prefill out-proj becomes column-parallel (AG+matmul instead of fp32 matmul+RS) MERGED 09-23 prefill, all 48 GDN layers THIRD-PARTY (TP=4, 4x p300c): TTFT -9.0% at 32k (5.86 -> 5.32 s), -10% at 16k. ESTIMATE for us: prefill -4..-9%, so makespan -3..-7% at B=8/32k S-M. Python only (3 files). The prod image already ships ttnn.experimental.all_gather_minimal_matmul_async (our tp_common.all_gather_matmul_prefill calls it), so this can be ported onto the bind-mounted patches/prod-local-metal/ tree with no image rebuild. It needs new out.colpar cache tensors (partial COLD).
2 #55444 DRAM-sharded decode matmul: in0_block_w may span several activation shards MERGED 09-14 decode, every DRAM-sharded projection THIRD-PARTY: Qwen3-8B P150x4 decode -7.6%, Llama-3.2-1B -30%, big models ~0%. ESTIMATE: our down-proj progcfg is in0_block_w=1, per_core_N=10 (bench/runs/n1n2-20260912/n2-raw.jsonl), i.e. about 272 serialized multicast hops. That puts it far below the PR's ~100-tiles-per-hop crossover, so hops are mostly exposed. Decode could drop 3-10 ms/token at B=8, which is makespan -1..-4% (more at 4k/8k) M. C++ (needs image rebuild) plus a qwen36 config change (widen in0_block_w). Measure one down-proj shape with a microbenchmark before building anything. This is not the closed "grid tuning" line: QWEN36_1D_GRID_* moved core counts, while this changes the hop count, a knob that did not exist before 09-14.
3 GDN chunked-prefill kernel stack: #53790 scan multicast (MERGED 09-16), #57440 fused prep->scan (open draft), #57445 SFPU T_inv (open draft, stacked), #53804 nbuf=2 (open draft) mixed prefill, chunk_gated_delta_rule THIRD-PARTY (TP=4): scan device time -35%, TTFT -2.8% (#53790). Fused op 1.52x at BH=16 and 1.23x at BH=32; we sit at BH=24, so ~1.3x; 32k TTFT -3.6% on top of #57377. ESTIMATE for us: prefill -3..-8% combined M. C++ rebuild. Our patches/tt-metal/0004-chunk-gdn-phased-cb-alignment touches the same chunk_gdn_phased* files, so expect conflicts. The fused path's cost model is calibrated on an 11x10 grid (QB2), and P150 also appears to be 11x10 (#57507 counts 22 cores in 2 rows). Take #53790 now; wait for #57440 to merge.
4 Blackhole SDPA roofline set: #57395 (consolidates #56937/#57016/#56897/#56989/#56925/#57170/...) open draft prefill SDPA (16 attention layers). Small decode gain via #57170 bf8 V-on-NoC1 THIRD-PARTY (p100a, 32q/8kv, d128, bfp8, HiFi2): causal prefill q128/k128 -19% (S1k) to -40% (S8k). ESTIMATE for us: prefill -2..-6% at 32k. Much more on the parked 128k/256k ladder, where SDPA is the 68% growth term M-L. C++. It overlaps our 0020-qwen36-sdpa-streaming-kernel-for-chunked-prefill. Unverified for our shape (GQA 12q/2kv per device, d=256, paged chunked SDPA). Wait for merge, then A/B against 0020.
5 MTP refresh: #55548 head 113099f0 (we carry 96f3f041), plus the plugin spec-decode contract (vllm-tt-plugin#118/#128 merged; #125/#127/#130/#131 open) and the Gemma4 serving pattern #56048 open (CHANGES_REQUESTED) decode, conc-1 THIRD-PARTY: 2.62x at ISL 128 on QB2 B=1. The new commits trace the drafter chain + reseed and shard the drafter argmax. The Gemma4 "adaptive block-output" model (spec at solo, plain batched at conc>1) is the ready-made answer to "MTP is B=1". No effect on B=4/8 makespan unless it runs in adaptive mode L. It serves the owner's MTP directive (0aaaaaaaaaaaaaaaaaa), not the makespan metric.

Two cheap items that can ride along with any rebuild:

  • #53627 dual-RISC ROW_MAJOR argmax (MERGED 09-17). It is bit-identical including tie-breaking, and it is exactly our 0014 path (to_layout(ROW_MAJOR) -> ttnn.argmax(dim=-1)). THIRD-PARTY: 1.64x median, measured on Wormhole. ESTIMATE: 0.3-1 ms/token, and it only applies if the op takes the multicore path.
  • num_workers_per_dram_bank=2 (#54242, already in the prod image; #55080 shows the recipe). This is a config-only A/B. The expected gain is low, because our readers already measure 454-469 GB/s (~90%) and that is the utilisation the PR reaches.

If we bump the pin past 2026-09-22, we must also carry #57507 (open). #57053 changed the reduce-scatter worker heuristic. qwen36 tp_common.matmul_reduce_scatter_prefill now TT_FATALs on a P150 (36 workers chosen, only 22 cores below the (0,8) offset) at M >= 288 tokens per chunk. That is our exact prefill path.

Our pins (as found)

Tree Upstream base Carried on top Used by
Production image vllm-tt:k2 (fork dhavli/tt-metal cfx/qwen36-p300-tp2 @ 3b31bad4, wheel 0.79.0.dev20260903+cfx.g3b31bad4) tt-metal main @ d2f4b3af (2026-09-02) (merge-base via GitHub compare) #53587 fused T=1 GDN decode (grafted), #53314, #53319, the FIR-tap edit. Python bind-mounted from patches/prod-local-metal/ + patches/prod-local-plugin/ serving, sweep-32k
Nix tt-metal/tt-metal-full = /home/orchestra/ttm-build/src @ 76983d00 main @ 60053f1f (2026-09-12) + #55548 @ 96f3f041 2 Tracy build commits + 24576047 lazy FP8 MTP/Tracy builds, sim
patches/tt-metal/0001-0023 applied onto the trees above TP=2 tuning, fused GDN inplace, device argmax (0014), hybrid KV spec (0015), prefix snapshots/traced chunk prefill (0017-0019), SDPA streaming (0020), async merge guards (0021-0022), resident decode + on-device RoPE (0023)
ttsim 0d22394c pins move as a set with tt-metal
vllm-tt-plugin c127c17d (2026-08-14) 0001-0006 (greedy gate, hybrid spec, prefix consumer, GDN snapshots, LRU, prefill trace) + bind-mounted model_runner.py
tt-inference-server 84fbb665 (2026-08-12) 0001 p150x2 device type

At the time of writing, main is 525 commits past 60053f1f and roughly 1,100 commits past the production base.

Correction to docs/UPSTREAM-WATCH-PLUGIN-METAL.md §1. That section says 743890db (#55343, the prerequisite for plugin#126) is not an ancestor of our tree. /home/orchestra/ttm-build/src is a shallow clone (git rev-parse --is-shallow-repository returns true), so merge-base --is-ancestor gives false negatives there. The GitHub compare API says 743890db, 722b1eb6 (#55689) and 172f60a8 (#54992) are in 60053f1f. They are genuinely absent from the 09-02 production base.

Merged upstream items that matter

"Local" means the 09-12 Nix tree; "prod" means the 09-02 image base.

PR Merged What it changes Why it matters to us Applies to 1x2 P150 TP=2? Expected impact In local / prod? Adopt
#57377 GDN out-proj column-parallel 09-23 Prefill out-proj uses AG+matmul in bf16 instead of matmul + fp32 RS. Default at TP>1, no env opt-out. Decode unchanged Prefill, 48 layers Yes. Measured at TP=4. At TP=2 the extra weight copy is [6144,2560] bfp8 x48 ≈ +764 MiB/device (the reason we HELD #56759) THIRD-PARTY -9% TTFT at 32k (TP=4); ESTIMATE prefill -4..-9% no / no Port to prod-local-metal behind our own env gate (on at <=32k, off for the parked 8x128k shape, where the 3.6 GiB margin binds)
#55444 wide in0_block_w for DRAM-sharded matmul 09-14 Sender gathers several activation shards per multicast block, which cuts the serialized hop count Decode MLP/QKV/out projections at M<=32 Yes (BH P150/P300 measured) THIRD-PARTY Qwen3-8B -7.6% decode; ESTIMATE 3-10 ms/token for us no / no Cherry-pick (+392/-29) or bump. Collides with the Metal 2.0 factory ports (#55961, #56114) on the prod base. Then widen in0_block_w in qwen36 mlp.py/attention/gdn configs
#53790 GDN scan per-head multicast 09-16 Shared scan inputs read once from DRAM and multicast; -70% read volume Prefill GDN scan Yes (op is geometry-generic; TP=2 is BH=24/chip) THIRD-PARTY scan -35%, TTFT -2.8% (TP=4) no / no C++ cherry-pick; check against 0004
#57368 chunk_gdn Qwen head-geometry tests 09-24 Adds the HV=48 / TP=2 geometry (24 heads/chip) to the reference test Correctness gate for any GDN kernel change we take Yes, it now covers our exact per-chip geometry test only no / no Take with any GDN kernel change
#56471 qwen36 host transposes only on cache miss 09-16 Skips materialising 27B params when the tensor cache hits Boot time (warm), not steady state Yes Faster warm boots (THIRD-PARTY: CI timeouts at 1200 s) no / no Python; easy port to prod-local-metal
#53627 dual-RISC ROW_MAJOR argmax 09-17 Splits the row loop across both DM RISCs; bit-identical ties Our device-argmax decode tail (0014) Yes, if the multicore path is taken THIRD-PARTY 1.64x (WH); ESTIMATE 0.3-1 ms/token no / no Rides along a rebuild
#53618 index tile built via NoC 09-18 Faster index tile for topk/sort/sampling/argmax Small decode-tail gain Yes small no / no Rides along
#54565 gate async-ahead token keep on supports_async_decode 09-14 tt_transformers Generator.decode_forward no longer adopts stale device tokens when async is off Correctness: observed on Qwen3.6-27B P150x4 with on-device sampling (8/32 requests truncated at token 2) Check whether our qwen36 path goes through this generator code (resident decode 0023 may bypass it) correctness no / no Audit, then port if reachable
#56225 SDPA decode correction fits FP32 half-sync DST 09-14 Fixes out-of-bounds DST slot with fp32 acc when a second core contributes (k_chunk 32 on BH) Possibly the root cause behind our "k_chunk_size=32 corrupts SDPA" trap; decode with fp32 acc Yes (BH) correctness no / no Take on bump; re-test the k_chunk=32 trap
#57180 SDPA honours accurate exp mode 09-23 exp_approx_mode=False previously still used the approximate exp Numerics and perf change for any SDPA call that sets exp_approx_mode=False Yes may be slightly slower and more accurate no / no Grep our SDPA program configs before a bump
#57126, #56258, #54992 host dispatch overhead (sub-device cache, prefill op RT args, MeshWorkload) 09-23 / 09-17 / 09-08 Lower host enqueue cost Only untraced paths. Our decode and prefill chunks are traced, so ≈0 Generic ESTIMATE ≈0 traced; helps warmup/capture #54992 local only Comes free with a bump
#55689 traced chunked-prefill RoPE slice fix (tt_transformers) 09-11 Correct RoPE width per trace bucket >=4096 Pattern check for our 0018 traced chunk prefill (qwen36 has its own RoPE path) Audit correctness local yes / prod no Audit 0018 for the same num_devices-derived slice width
#55343 trace allocation fixes 09-11 Prerequisite of plugin#126 Unblocks a plugin bump on the local tree Generic enabling local yes / prod no none needed on local
#53430 indexed fused paged KV update 09-01 Writes a variable number of accepted K/V rows to arbitrary pages MTP verify/commit (variable accepted count) Yes enabling for MTP at B>1 local yes / prod yes Use in the MTP work
#56048 Gemma4 TT-native spec serving (MTP+dFlash) 09-21 Spec-decode model classes plus adaptive block-output (spec when solo, batched otherwise) Template for ThinkingCap MTP under vLLM Pattern (P150x8 / Galaxy) THIRD-PARTY MTP ~2.6x at conc-1 no / no Read and copy the pattern
#53917 Qwen3.6-35B-A3B MoE 09-16 Large refactor inside models/demos/blackhole/qwen36 Conflict source for every prod-local-metal file n/a none no / no Expect rebase pain on bump
#57053 RS/AG heuristic change 09-22 RS workers-per-link retuned on 8-device BH Regression: FATAL in qwen36 prefill on P150 until #57507 lands Yes, it breaks us breaking no / no Carry #57507
#48280 -> #57434 fabric express link routing, then revert 09-21 -> 09-23 enabled, then reverted Avoid pinning inside that window
#53557 -> #56467 stable topk, then sampling fix 09-12 -> 09-17 Stable topk introduced non-finite sampled logprobs; fixed 09-17 Only host/device non-greedy sampling Avoid pinning inside that window
#57517 rms_allgather hang fix 09-23 Restores the re-init dropped by #55697 Only if we use rms_allgather (Galaxy path) probably no

Open PRs to watch or cherry-pick

PR Updated What Relevance to us 1x2 P150? THIRD-PARTY numbers Adopt
#57507 RS default workers respect core_grid_offset 09-24 Fixes the #57053 FATAL Mandatory with any bump past 09-22 Yes (P150, 22 cores) fix cherry-pick (+50/-11)
#57440 fused chunk_gated_delta_rule (draft) 09-24 prep+scan in one program, no DRAM intermediates; bit-exact vs phased; QWEN_GDN_PATH override Prefill GDN Yes; grid-generic, calibrated on 11x10 1.94x op at BH=12, 1.23-1.52x at BH=16-32; 32k TTFT -3.6% (TP=4) wait for merge, then build
#57445 SFPU forward-substitution T_inv (draft, stacked on #57440) 09-24 Default on BH at chunk 32; QWEN_GDN_TINV pin Prefill GDN Yes -2.5..-13% on the fused op; model logits PCC 0.9999 with #57440
#53804 scan nbuf=2 (draft) 09-16 Prefetch gated to mcast && NV>=4 Prefill GDN Depends on the NV we get at BH=24 +15% scan at NV=4 low
#57395 BH SDPA roofline consolidation (draft) 09-23 K/V chains, streaming fp32 DEST, exp fix, bf8 V on NoC1 Prefill attention (growth term) + small decode Yes (BH); d=256 GQA untested -19..-40% causal prefill op (p100a) wait; A/B vs 0020
#57170 sdpa decode: V on 2nd NoC for bfp8/bfp4 K/V (in #57395) 09-22 Splits the K/V request stream across both NoCs Decode attention with bf8 KV (our default) Yes -3.1% at bfp8 32k cache (op) with #57395
#57266 / #57267 / #56989 BH streaming SDPA precision recipes 09-21/22 Streaming kernel with fp32 DEST Overlaps our 0020 Yes track
#56898 / #56985 / #56942 / #57397 tt_transformers BH SDPA grid/rope configs (drafts) 09-18/23 Full BH grid for SDPA prefill/decode Config ideas only; qwen36 already uses the full BH grid for chunked SDPA Yes read
#54106 fused greedy-argmax epilogue on DRAM-sharded LM-head matmul (draft, stacked on #54105 RVV argmax) 08-31 Argmax partials computed on the pack RISC's vector unit during the LM-head matmul Would remove our to_layout+argmax tail entirely; names qwen36 explicitly BH only; needs a TP=2 combine THIRD-PARTY tails 0.92-4.4 ms/token removed watch; stale since 08-31
#55214 small_m_matmul (draft) 09-02 DRAM-bound skinny matmul, 79-94% peak on P150 Alternative decode matmul; bf16 in/out only (our weights are bfp4/bfp8) partial watch
#57251 DMA writes on Blackhole 09-21 H2D via DMA instead of MMIO Faster weight upload at boot; tiny host->device transfers per step Yes not measured on BH low risk; take on bump
#57252 UMD bump + relaxed DRAM write ordering 09-21 IoOrdering change Unknown perf; UMD bump means firmware/KMD compatibility check Yes watch; check the tt-kmd pin
#44421 multithreaded AVX2 host BFP8 tilizer 09-10 Byte-identical, faster host bfp packing COLD cache build for ThinkingCap (bfp4/bfp8 conversion); cfx-llm2 has 4 cores, so modest Yes (host) nice-to-have for cold boots
#55465 kernel prewarm JIT capture 09-10 Pre-captures JIT kernels Boot time Yes watch
#56832 qwen36 reference-accuracy gate 09-24 text_demo.py -k accuracy_512, top-1/top-5 vs HF (98.63%/100% on P150x4) A ready accuracy gate for bf8 KV / GDN kernel / #57377 numerics changes (Qwen3.6 refpt, not ThinkingCap) Yes adopt the harness idea; generate a ThinkingCap refpt
#55548 MTP spec decode 09-24 +9403/-515; new commits trace the drafter chain + reseed, shard the drafter argmax, and address review MTP directive Built on QB2 B=1; our tree already merges it 2.62x @ ISL 128 re-merge head 113099f0 into the MTP tree
#56939 Llama prefill: scope cache gather to the prefix 09-24 Per-chunk cost follows the prompt, not the allocated cache Pattern check: does our chunked SDPA read max_model_len-sized ranges? The sweep holds max_model_len=34816 fixed pattern 2.6x -> 1.0x capacity penalty (Galaxy) audit only
#56184 device RNG for trace replay; #55892 on-device grammar sampling 09-23/16 Non-greedy sampling on device Only if we leave greedy later
#53587 / #53482 fused T=1 GDN decode 09-02 (stale) Already grafted into prod nothing to do
#56119 Qwen3.8-27B QB2 opt (draft) 09-10 (stale) Already audited: refused for TP=2 nothing to do

vllm-tt-plugin (our pin c127c17d, 2026-08-14)

  • Merged since the pin. #119 interleaves decode steps between prefill chunks. It is default ON (2 prefill steps : 1 decode step). THIRD-PARTY: worst inter-token gap 11152 -> 899 ms, +1.0% median TTFT, measured on T3K. For our batch-makespan metric it is roughly zero-sum. A TT step is prefill-only or decode-only, so total work does not shrink, and the last agent's TTFT gets slightly worse. It helps interactivity, not makespan, so do not count it as a win on sweep-32k. Also merged:
    • #126 reverts the decode-first warmup. Prod is already immune.
    • #79 gates chunked prefill on a capability.
    • #78 async decode reload planning.
    • #118 adaptive spec block-output.
    • #120 / #128 spec contract and host accept walk.
    • #135 scheduler cleanup across prefill/decode fallback.
  • Open.
    • Spec decode chain: #125 config admit, #127 [B,1+K] block, #130 verify-then-propose, #131 on-device drafter. This is the vLLM side ThinkingCap MTP needs.
    • #114 host fallback for greedy-only device samplers overlaps our 0001.
    • #22 MambaSpec state ownership (stale draft) overlaps our 0002-0005 prefix work.
    • #101 vLLM 0.28 (draft).
  • Adopt. A plugin bump is now possible on the local tree (#55343 is present). On prod it needs the metal bump first. Our 6 plugin patches plus the bind-mounted model_runner.py will conflict with #118/#119/#135 in scheduler.py/model_runner.py.

tenstorrent/vllm and tt-inference-server

  • tenstorrent/vllm. The default branch main was last pushed 2026-09-16. I found no dev branch and no Qwen/TT activity. The plugin is the live integration point, so there is nothing to take here.
  • tt-inference-server (our pin is 08-12).
    • Qwen3.6-27B appears only as a P300X2 impl=quetzal row (#5147; a generated implementation with max_context 8192 and concurrency 1) plus the existing qwen36_blackhole row.
    • Gemma4 MTP/dFlash spec serving specs (#5137) and --spec-decode-metrics-url (#4978) are useful when we instrument MTP acceptance.
    • P150X4 was dropped from nightly CI (#5220). That means less upstream CI coverage for P150 multi-card, not more.
    • No performance items.

Recommended action

A selective path, not a wholesale pin bump. The production image is on a 09-02 base with grafted open PRs, and main has since landed a qwen36 MoE refactor, Metal 2.0 ports of the matmul/SDPA/paged-cache factories, and a P150-breaking RS heuristic change. A bump means rebasing all 23 metal patches plus 5 prod-local files, and it moves ttsim with it. Order:

  1. Now, Python only, no rebuild. Port #57377 into patches/prod-local-metal/.../gdn/tp.py + tp_common.py, behind QWEN36_GDN_OUT_COLPAR (default on for max_model_len <= 34816, off for the parked long shapes). Build the out.colpar cache once. Also port #56471 (warm boot). Audit #54565 and #55689 against our generator and 0018.
  2. Probe before building. Run a single-op microbenchmark of our down-proj / gate-up / QKV DRAM-sharded matmuls at M=32 with in0_block_w in {1, 2, 4, 8} on a #55444 build. Run the num_workers_per_dram_bank=2 A/B on the current image (config only). Promote #55444 only if the op time moves.
  3. Next image rebuild (on nixos, not cfx-llm2), cherry-picked onto the prod fork: #55444, #53790 (+ #57368 tests), #53627, #53618. Resolve against 0004 and the grafted #53587. Carry #57507 pre-emptively.
  4. Wait for merge: #57440/#57445 and #57395. Then either do a second cherry-pick round or, if the list keeps growing, do one coordinated bump (tt-metal + ttsim together) to a main commit that is after #57507 merges and outside the #48280 and #53557 windows. Re-merge #55548 head in that bump.
  5. MTP track (parallel). Re-merge #55548 @ 113099f0. Use #53430 for variable-row KV commit, the #56048 adaptive block-output pattern, and the plugin spec contract (#118/#128, then #125-#131).

Validation plan, for every adopted item, in this order:

  1. Op/layer PCC gates. test_gdn_tp.py (#57377 adds test_gdn_out_agmm_vs_mmrs + skew determinism), test_chunk_gated_delta_rule.py with the new HV=48/TP=2 geometry (#57368), and the SDPA prefill/decode unit tests at our shape (12q/2kv, d256, bf8 KV).
  2. Fixed-width greedy equivalence at B=4 and B=8 against the current reference (bench/runs/batch-equiv-ref-*, bucketing off). #57377 and #57445 are not bit-exact: they are PCC-class numerics changes, so expect the hash to move. Use the accuracy gate (Gate A bench/kv-accuracy.py at 16k/32k plus GSM8K / infbench stage 3) instead of demanding sha identity. #53790, #53627 and fused-vs-phased are claimed bit-exact, so require sha identity for those.
  3. Arm self-identification. Add a banner per new lever (like the 0016 QWEN_SDPA_BF8 banner) so every record proves its arm.
  4. Performance. bench/ab/sweep-32k.sh, 4k/8k/16k/32k x B={4,8}, current optimized arm vs arm + lever, one source tree differing only by env, max_model_len 34816. Rank by makespan, and report TTFT and decode ms/token alongside. Report PP tok/s and TG ms/token in every status (owner rule). For decode-side items (#55444, #53627), the B=4 control decode must stay faster than the B=8 baseline (existing tell).
  5. Cache arm. Re-run bench/ab/cache-speed-16k-32k.sh for prefill-side items, because prefix caching changes which prefill tokens are actually computed.
  6. Record everything in bench/runs/*.jsonl with the upstream PR number and commit in the config.

Risks

  • #57053 P150 FATAL. A bump past 09-22 without #57507 cannot run qwen36 prefill on P150 at chunk >= 288 tokens.
  • #57377 memory. It is on by default with no opt-out. It adds ~+764 MiB/device weights at TP=2 (partly offset by dropped fp32 RS trace buffers, -97 MiB at TP=4) and cuts ~21% of the 3.6 GiB margin at 8x128k. This is fine at the <=32k benchmark. Gate it ourselves.
  • Numerics changes that break sha-equivalence by design: #57377 (bf16 gate output before matmul), #57445 (SFPU T_inv), #57180 (accurate exp now really accurate), #56225 (SDPA decode correction). Decide per item which bar applies before the run, not after.
  • Rebase load. #53917 (qwen36 MoE refactor) and the Metal 2.0 factory ports (matmul DRAM-sharded #55961/#56114/#56761/#56836, sdpa #54468, sdpa_decode #54249, paged_cache #56022) collide with 0004, 0020, 0023 and the grafted #53587/#53314/#53319.
  • Coverage mismatch. Almost all upstream qwen36 numbers are TP=4 (P150x4 bh_quietbox_2, or 4x p300c QB2). Only #53587 (stale) targets 1x2. TP=2 has twice the heads per chip (BH=24), so every kernel cost model (fused GDN geometry, SDPA chain gates) runs outside its calibrated points.
  • Pinning windows to avoid: 09-21..09-23 (fabric express routing, reverted), 09-12..09-17 (stable-topk non-finite logprobs), and before #57517 if rms_allgather is ever used.
  • UMD bump (#57252). It may require a matching tt-kmd / firmware. Firmware updates are never unattended.
  • Shallow clone. Do not use local git merge-base on /home/orchestra/ttm-build/src for "is X in our tree" questions; use the GitHub compare API (see skill in-tree.sh).

Sources

  • Our repo: flake.nix/flake.lock, patches/{tt-metal,prod-local-metal,prod-local-plugin,vllm-tt-plugin,upstream}/, patches/PROD-SERVING-SOURCES.md, docs/UPSTREAM-WATCH-PLUGIN-METAL.md, docs/CXX-BUILD.md, docker/Dockerfile.vllm-tt-k2*, docs/DIRECTIVES-LEDGER.md, bench/runs/n1n2-20260912/n2-raw.jsonl, docs/TRACY-PREFILL-PROFILE.md.
  • GitHub API (gh, authenticated): repos/tenstorrent/tt-metal/commits?sha=main&since=2026-08-28 (1,104 commits, 326 after keyword filter), path-scoped commit lists for models/demos/blackhole/qwen36, …/transformer/sdpa, …/operations/experimental; open-PR keyword search (228 PRs); compare/<c>...<base> for ancestry; PR bodies for every item above.
  • vllm-tt-plugin commits since 2026-09-01 and its open PRs; tt-inference-server commits since 2026-08-20; tenstorrent/vllm repo metadata.
This is desk research only. Nothing was run on cfx-llm2. Every number below is either **THIRD-PARTY** (from the PR itself, on other hardware) or **ESTIMATE** (my arithmetic). None of them is measured on our 2x P150a TP=2 box. Our own memory note says extrapolations like these have been wrong by 49-105%, so treat each one as a reason to measure, not as a result. Scope: Qwen3.8-27B / ThinkingCap-27B-FP8 (48 GDN + 16 full-attention layers), 1x2 P150a mesh, TP=2, vLLM. The metric is batch makespan and TTFT at B=4/8 with <=32k in / <=1k out. Prefill is 74-79% of wall at those shapes (sweep-32k, 2026-09-14). ## TL;DR: top 5 to act on (ranked by expected makespan/TTFT effect ÷ effort) | # | Item | Status | Where it acts | Expected effect | Effort | |---|---|---|---|---|---| | 1 | **[#57377](https://github.com/tenstorrent/tt-metal/pull/57377) GDN prefill out-proj becomes column-parallel (AG+matmul instead of fp32 matmul+RS)** | MERGED 09-23 | prefill, all 48 GDN layers | THIRD-PARTY (TP=4, 4x p300c): TTFT -9.0% at 32k (5.86 -> 5.32 s), -10% at 16k. ESTIMATE for us: prefill -4..-9%, so **makespan -3..-7%** at B=8/32k | **S-M**. Python only (3 files). The prod image already ships `ttnn.experimental.all_gather_minimal_matmul_async` (our `tp_common.all_gather_matmul_prefill` calls it), so this can be ported onto the bind-mounted `patches/prod-local-metal/` tree with no image rebuild. It needs new `out.colpar` cache tensors (partial COLD). | | 2 | **[#55444](https://github.com/tenstorrent/tt-metal/pull/55444) DRAM-sharded decode matmul: `in0_block_w` may span several activation shards** | MERGED 09-14 | decode, every DRAM-sharded projection | THIRD-PARTY: Qwen3-8B P150x4 decode -7.6%, Llama-3.2-1B -30%, big models ~0%. ESTIMATE: our down-proj progcfg is `in0_block_w=1, per_core_N=10` (`bench/runs/n1n2-20260912/n2-raw.jsonl`), i.e. about 272 serialized multicast hops. That puts it far below the PR's ~100-tiles-per-hop crossover, so hops are mostly exposed. Decode could drop 3-10 ms/token at B=8, which is **makespan -1..-4%** (more at 4k/8k) | **M**. C++ (needs image rebuild) plus a qwen36 config change (widen `in0_block_w`). Measure one down-proj shape with a microbenchmark before building anything. **This is not the closed "grid tuning" line**: `QWEN36_1D_GRID_*` moved core counts, while this changes the hop count, a knob that did not exist before 09-14. | | 3 | **GDN chunked-prefill kernel stack**: [#53790](https://github.com/tenstorrent/tt-metal/pull/53790) scan multicast (MERGED 09-16), [#57440](https://github.com/tenstorrent/tt-metal/pull/57440) fused prep->scan (open draft), [#57445](https://github.com/tenstorrent/tt-metal/pull/57445) SFPU T_inv (open draft, stacked), [#53804](https://github.com/tenstorrent/tt-metal/pull/53804) nbuf=2 (open draft) | mixed | prefill, `chunk_gated_delta_rule` | THIRD-PARTY (TP=4): scan device time -35%, TTFT -2.8% (#53790). Fused op 1.52x at BH=16 and 1.23x at BH=32; we sit at BH=24, so ~1.3x; 32k TTFT -3.6% on top of #57377. ESTIMATE for us: **prefill -3..-8%** combined | **M**. C++ rebuild. Our `patches/tt-metal/0004-chunk-gdn-phased-cb-alignment` touches the same `chunk_gdn_phased*` files, so expect conflicts. The fused path's cost model is calibrated on an 11x10 grid (QB2), and P150 also appears to be 11x10 (#57507 counts 22 cores in 2 rows). Take #53790 now; wait for #57440 to merge. | | 4 | **Blackhole SDPA roofline set**: [#57395](https://github.com/tenstorrent/tt-metal/pull/57395) (consolidates #56937/#57016/#56897/#56989/#56925/#57170/...) | open draft | prefill SDPA (16 attention layers). Small decode gain via #57170 bf8 V-on-NoC1 | THIRD-PARTY (p100a, 32q/8kv, d128, bfp8, HiFi2): causal prefill q128/k128 -19% (S1k) to -40% (S8k). ESTIMATE for us: **prefill -2..-6% at 32k**. Much more on the parked 128k/256k ladder, where SDPA is the 68% growth term | **M-L**. C++. It overlaps our `0020-qwen36-sdpa-streaming-kernel-for-chunked-prefill`. Unverified for our shape (GQA 12q/2kv per device, **d=256**, paged chunked SDPA). Wait for merge, then A/B against 0020. | | 5 | **MTP refresh**: [#55548](https://github.com/tenstorrent/tt-metal/pull/55548) head `113099f0` (we carry `96f3f041`), plus the plugin spec-decode contract ([vllm-tt-plugin#118](https://github.com/tenstorrent/vllm-tt-plugin/pull/118)/[#128](https://github.com/tenstorrent/vllm-tt-plugin/pull/128) merged; [#125](https://github.com/tenstorrent/vllm-tt-plugin/pull/125)/[#127](https://github.com/tenstorrent/vllm-tt-plugin/pull/127)/[#130](https://github.com/tenstorrent/vllm-tt-plugin/pull/130)/[#131](https://github.com/tenstorrent/vllm-tt-plugin/pull/131) open) and the Gemma4 serving pattern [#56048](https://github.com/tenstorrent/tt-metal/pull/56048) | open (CHANGES_REQUESTED) | decode, conc-1 | THIRD-PARTY: 2.62x at ISL 128 on QB2 B=1. The new commits trace the drafter chain + reseed and shard the drafter argmax. The Gemma4 "adaptive block-output" model (spec at solo, plain batched at conc>1) is the ready-made answer to "MTP is B=1". **No effect on B=4/8 makespan** unless it runs in adaptive mode | **L**. It serves the owner's MTP directive (0aaaaaaaaaaaaaaaaaa), not the makespan metric. | Two cheap items that can ride along with any rebuild: - [#53627](https://github.com/tenstorrent/tt-metal/pull/53627) dual-RISC ROW_MAJOR argmax (MERGED 09-17). It is bit-identical including tie-breaking, and it is exactly our 0014 path (`to_layout(ROW_MAJOR) -> ttnn.argmax(dim=-1)`). THIRD-PARTY: 1.64x median, measured on Wormhole. ESTIMATE: 0.3-1 ms/token, and it only applies if the op takes the multicore path. - `num_workers_per_dram_bank=2` ([#54242](https://github.com/tenstorrent/tt-metal/pull/54242), already in the prod image; [#55080](https://github.com/tenstorrent/tt-metal/pull/55080) shows the recipe). This is a config-only A/B. The expected gain is low, because our readers already measure 454-469 GB/s (~90%) and that is the utilisation the PR reaches. **If we bump the pin past 2026-09-22, we must also carry [#57507](https://github.com/tenstorrent/tt-metal/pull/57507) (open).** [#57053](https://github.com/tenstorrent/tt-metal/pull/57053) changed the reduce-scatter worker heuristic. `qwen36 tp_common.matmul_reduce_scatter_prefill` now TT_FATALs on a P150 (36 workers chosen, only 22 cores below the `(0,8)` offset) at M >= 288 tokens per chunk. That is our exact prefill path. ## Our pins (as found) | Tree | Upstream base | Carried on top | Used by | |---|---|---|---| | **Production image** `vllm-tt:k2` (fork `dhavli/tt-metal` `cfx/qwen36-p300-tp2` @ `3b31bad4`, wheel `0.79.0.dev20260903+cfx.g3b31bad4`) | tt-metal main **@ `d2f4b3af` (2026-09-02)** (merge-base via GitHub compare) | #53587 fused T=1 GDN decode (grafted), #53314, #53319, the FIR-tap edit. Python bind-mounted from `patches/prod-local-metal/` + `patches/prod-local-plugin/` | serving, sweep-32k | | **Nix `tt-metal`/`tt-metal-full`** = `/home/orchestra/ttm-build/src` @ `76983d00` | main **@ `60053f1f` (2026-09-12)** + #55548 @ `96f3f041` | 2 Tracy build commits + `24576047` lazy FP8 | MTP/Tracy builds, sim | | `patches/tt-metal/0001-0023` | applied onto the trees above | TP=2 tuning, fused GDN inplace, device argmax (0014), hybrid KV spec (0015), prefix snapshots/traced chunk prefill (0017-0019), SDPA streaming (0020), async merge guards (0021-0022), resident decode + on-device RoPE (0023) | | | ttsim | `0d22394c` | | pins move as a set with tt-metal | | vllm-tt-plugin | `c127c17d` (**2026-08-14**) | 0001-0006 (greedy gate, hybrid spec, prefix consumer, GDN snapshots, LRU, prefill trace) + bind-mounted `model_runner.py` | | | tt-inference-server | `84fbb665` (2026-08-12) | 0001 p150x2 device type | | At the time of writing, main is **525 commits past `60053f1f`** and roughly 1,100 commits past the production base. **Correction to `docs/UPSTREAM-WATCH-PLUGIN-METAL.md` §1.** That section says `743890db` (#55343, the prerequisite for plugin#126) is not an ancestor of our tree. `/home/orchestra/ttm-build/src` is a **shallow clone** (`git rev-parse --is-shallow-repository` returns `true`), so `merge-base --is-ancestor` gives false negatives there. The GitHub compare API says `743890db`, `722b1eb6` (#55689) and `172f60a8` (#54992) **are** in `60053f1f`. They are genuinely absent from the 09-02 production base. ## Merged upstream items that matter "Local" means the 09-12 Nix tree; "prod" means the 09-02 image base. | PR | Merged | What it changes | Why it matters to us | Applies to 1x2 P150 TP=2? | Expected impact | In local / prod? | Adopt | |---|---|---|---|---|---|---|---| | [#57377](https://github.com/tenstorrent/tt-metal/pull/57377) GDN out-proj column-parallel | 09-23 | Prefill out-proj uses AG+matmul in bf16 instead of matmul + fp32 RS. **Default at TP>1, no env opt-out.** Decode unchanged | Prefill, 48 layers | Yes. Measured at TP=4. At TP=2 the extra weight copy is `[6144,2560]` bfp8 x48 ≈ **+764 MiB/device** (the reason we HELD #56759) | THIRD-PARTY -9% TTFT at 32k (TP=4); ESTIMATE prefill -4..-9% | no / no | Port to prod-local-metal behind our own env gate (on at <=32k, off for the parked 8x128k shape, where the 3.6 GiB margin binds) | | [#55444](https://github.com/tenstorrent/tt-metal/pull/55444) wide `in0_block_w` for DRAM-sharded matmul | 09-14 | Sender gathers several activation shards per multicast block, which cuts the serialized hop count | Decode MLP/QKV/out projections at M<=32 | Yes (BH P150/P300 measured) | THIRD-PARTY Qwen3-8B -7.6% decode; ESTIMATE 3-10 ms/token for us | no / no | Cherry-pick (+392/-29) or bump. Collides with the Metal 2.0 factory ports (#55961, #56114) on the prod base. Then widen `in0_block_w` in qwen36 `mlp.py`/attention/gdn configs | | [#53790](https://github.com/tenstorrent/tt-metal/pull/53790) GDN scan per-head multicast | 09-16 | Shared scan inputs read once from DRAM and multicast; -70% read volume | Prefill GDN scan | Yes (op is geometry-generic; TP=2 is BH=24/chip) | THIRD-PARTY scan -35%, TTFT -2.8% (TP=4) | no / no | C++ cherry-pick; check against 0004 | | [#57368](https://github.com/tenstorrent/tt-metal/pull/57368) chunk_gdn Qwen head-geometry tests | 09-24 | Adds the HV=48 / TP=2 geometry (24 heads/chip) to the reference test | Correctness gate for any GDN kernel change we take | Yes, it now covers our exact per-chip geometry | test only | no / no | Take with any GDN kernel change | | [#56471](https://github.com/tenstorrent/tt-metal/pull/56471) qwen36 host transposes only on cache miss | 09-16 | Skips materialising 27B params when the tensor cache hits | Boot time (warm), not steady state | Yes | Faster warm boots (THIRD-PARTY: CI timeouts at 1200 s) | no / no | Python; easy port to prod-local-metal | | [#53627](https://github.com/tenstorrent/tt-metal/pull/53627) dual-RISC ROW_MAJOR argmax | 09-17 | Splits the row loop across both DM RISCs; bit-identical ties | Our device-argmax decode tail (0014) | Yes, if the multicore path is taken | THIRD-PARTY 1.64x (WH); ESTIMATE 0.3-1 ms/token | no / no | Rides along a rebuild | | [#53618](https://github.com/tenstorrent/tt-metal/pull/53618) index tile built via NoC | 09-18 | Faster index tile for topk/sort/sampling/argmax | Small decode-tail gain | Yes | small | no / no | Rides along | | [#54565](https://github.com/tenstorrent/tt-metal/pull/54565) gate async-ahead token keep on `supports_async_decode` | 09-14 | tt_transformers `Generator.decode_forward` no longer adopts stale device tokens when async is off | **Correctness**: observed on Qwen3.6-27B P150x4 with on-device sampling (8/32 requests truncated at token 2) | Check whether our qwen36 path goes through this generator code (resident decode 0023 may bypass it) | correctness | no / no | Audit, then port if reachable | | [#56225](https://github.com/tenstorrent/tt-metal/pull/56225) SDPA decode correction fits FP32 half-sync DST | 09-14 | Fixes out-of-bounds DST slot with fp32 acc when a second core contributes (k_chunk 32 on BH) | Possibly the root cause behind our "k_chunk_size=32 corrupts SDPA" trap; decode with fp32 acc | Yes (BH) | correctness | no / no | Take on bump; re-test the k_chunk=32 trap | | [#57180](https://github.com/tenstorrent/tt-metal/pull/57180) SDPA honours accurate exp mode | 09-23 | `exp_approx_mode=False` previously still used the approximate exp | **Numerics and perf change** for any SDPA call that sets `exp_approx_mode=False` | Yes | may be slightly slower and more accurate | no / no | Grep our SDPA program configs before a bump | | [#57126](https://github.com/tenstorrent/tt-metal/pull/57126), [#56258](https://github.com/tenstorrent/tt-metal/pull/56258), [#54992](https://github.com/tenstorrent/tt-metal/pull/54992) host dispatch overhead (sub-device cache, prefill op RT args, MeshWorkload) | 09-23 / 09-17 / 09-08 | Lower host enqueue cost | Only untraced paths. Our decode and prefill chunks are traced, so ≈0 | Generic | ESTIMATE ≈0 traced; helps warmup/capture | #54992 local only | Comes free with a bump | | [#55689](https://github.com/tenstorrent/tt-metal/pull/55689) traced chunked-prefill RoPE slice fix (tt_transformers) | 09-11 | Correct RoPE width per trace bucket >=4096 | Pattern check for our 0018 traced chunk prefill (qwen36 has its own RoPE path) | Audit | correctness | local yes / prod no | Audit 0018 for the same `num_devices`-derived slice width | | [#55343](https://github.com/tenstorrent/tt-metal/pull/55343) trace allocation fixes | 09-11 | Prerequisite of plugin#126 | Unblocks a plugin bump on the local tree | Generic | enabling | local yes / prod no | none needed on local | | [#53430](https://github.com/tenstorrent/tt-metal/pull/53430) indexed fused paged KV update | 09-01 | Writes a variable number of accepted K/V rows to arbitrary pages | MTP verify/commit (variable accepted count) | Yes | enabling for MTP at B>1 | local yes / prod yes | Use in the MTP work | | [#56048](https://github.com/tenstorrent/tt-metal/pull/56048) Gemma4 TT-native spec serving (MTP+dFlash) | 09-21 | Spec-decode model classes plus adaptive block-output (spec when solo, batched otherwise) | Template for ThinkingCap MTP under vLLM | Pattern (P150x8 / Galaxy) | THIRD-PARTY MTP ~2.6x at conc-1 | no / no | Read and copy the pattern | | [#53917](https://github.com/tenstorrent/tt-metal/pull/53917) Qwen3.6-35B-A3B MoE | 09-16 | Large refactor inside `models/demos/blackhole/qwen36` | **Conflict source** for every prod-local-metal file | n/a | none | no / no | Expect rebase pain on bump | | [#57053](https://github.com/tenstorrent/tt-metal/pull/57053) RS/AG heuristic change | 09-22 | RS workers-per-link retuned on 8-device BH | **Regression**: FATAL in qwen36 prefill on P150 until #57507 lands | Yes, it breaks us | breaking | no / no | Carry #57507 | | [#48280](https://github.com/tenstorrent/tt-metal/pull/48280) -> [#57434](https://github.com/tenstorrent/tt-metal/pull/57434) fabric express link routing, then revert | 09-21 -> 09-23 | enabled, then reverted | Avoid pinning inside that window | | | | | | [#53557](https://github.com/tenstorrent/tt-metal/pull/53557) -> [#56467](https://github.com/tenstorrent/tt-metal/pull/56467) stable topk, then sampling fix | 09-12 -> 09-17 | Stable topk introduced non-finite sampled logprobs; fixed 09-17 | Only host/device non-greedy sampling | | | | Avoid pinning inside that window | | [#57517](https://github.com/tenstorrent/tt-metal/pull/57517) rms_allgather hang fix | 09-23 | Restores the re-init dropped by #55697 | Only if we use `rms_allgather` (Galaxy path) | probably no | | | | ## Open PRs to watch or cherry-pick | PR | Updated | What | Relevance to us | 1x2 P150? | THIRD-PARTY numbers | Adopt | |---|---|---|---|---|---|---| | [#57507](https://github.com/tenstorrent/tt-metal/pull/57507) RS default workers respect `core_grid_offset` | 09-24 | Fixes the #57053 FATAL | **Mandatory with any bump past 09-22** | Yes (P150, 22 cores) | fix | cherry-pick (+50/-11) | | [#57440](https://github.com/tenstorrent/tt-metal/pull/57440) fused chunk_gated_delta_rule (draft) | 09-24 | prep+scan in one program, no DRAM intermediates; bit-exact vs phased; `QWEN_GDN_PATH` override | Prefill GDN | Yes; grid-generic, calibrated on 11x10 | 1.94x op at BH=12, 1.23-1.52x at BH=16-32; 32k TTFT -3.6% (TP=4) | wait for merge, then build | | [#57445](https://github.com/tenstorrent/tt-metal/pull/57445) SFPU forward-substitution T_inv (draft, stacked on #57440) | 09-24 | Default on BH at chunk 32; `QWEN_GDN_TINV` pin | Prefill GDN | Yes | -2.5..-13% on the fused op; model logits PCC 0.9999 | with #57440 | | [#53804](https://github.com/tenstorrent/tt-metal/pull/53804) scan nbuf=2 (draft) | 09-16 | Prefetch gated to `mcast && NV>=4` | Prefill GDN | Depends on the NV we get at BH=24 | +15% scan at NV=4 | low | | [#57395](https://github.com/tenstorrent/tt-metal/pull/57395) BH SDPA roofline consolidation (draft) | 09-23 | K/V chains, streaming fp32 DEST, exp fix, bf8 V on NoC1 | Prefill attention (growth term) + small decode | Yes (BH); d=256 GQA untested | -19..-40% causal prefill op (p100a) | wait; A/B vs 0020 | | [#57170](https://github.com/tenstorrent/tt-metal/pull/57170) sdpa decode: V on 2nd NoC for bfp8/bfp4 K/V (in #57395) | 09-22 | Splits the K/V request stream across both NoCs | Decode attention with **bf8 KV** (our default) | Yes | -3.1% at bfp8 32k cache (op) | with #57395 | | [#57266](https://github.com/tenstorrent/tt-metal/pull/57266) / [#57267](https://github.com/tenstorrent/tt-metal/pull/57267) / [#56989](https://github.com/tenstorrent/tt-metal/pull/56989) BH streaming SDPA precision recipes | 09-21/22 | Streaming kernel with fp32 DEST | Overlaps our 0020 | Yes | | track | | [#56898](https://github.com/tenstorrent/tt-metal/pull/56898) / [#56985](https://github.com/tenstorrent/tt-metal/pull/56985) / [#56942](https://github.com/tenstorrent/tt-metal/pull/56942) / [#57397](https://github.com/tenstorrent/tt-metal/pull/57397) tt_transformers BH SDPA grid/rope configs (drafts) | 09-18/23 | Full BH grid for SDPA prefill/decode | Config ideas only; qwen36 already uses the full BH grid for chunked SDPA | Yes | | read | | [#54106](https://github.com/tenstorrent/tt-metal/pull/54106) fused greedy-argmax epilogue on DRAM-sharded LM-head matmul (draft, stacked on [#54105](https://github.com/tenstorrent/tt-metal/pull/54105) RVV argmax) | 08-31 | Argmax partials computed on the pack RISC's vector unit during the LM-head matmul | Would remove our `to_layout+argmax` tail entirely; names qwen36 explicitly | BH only; needs a TP=2 combine | THIRD-PARTY tails 0.92-4.4 ms/token removed | watch; stale since 08-31 | | [#55214](https://github.com/tenstorrent/tt-metal/pull/55214) small_m_matmul (draft) | 09-02 | DRAM-bound skinny matmul, 79-94% peak on P150 | Alternative decode matmul; bf16 in/out only (our weights are bfp4/bfp8) | partial | | watch | | [#57251](https://github.com/tenstorrent/tt-metal/pull/57251) DMA writes on Blackhole | 09-21 | H2D via DMA instead of MMIO | Faster weight upload at boot; tiny host->device transfers per step | Yes | not measured on BH | low risk; take on bump | | [#57252](https://github.com/tenstorrent/tt-metal/pull/57252) UMD bump + relaxed DRAM write ordering | 09-21 | IoOrdering change | Unknown perf; UMD bump means firmware/KMD compatibility check | Yes | | watch; check the tt-kmd pin | | [#44421](https://github.com/tenstorrent/tt-metal/pull/44421) multithreaded AVX2 host BFP8 tilizer | 09-10 | Byte-identical, faster host bfp packing | **COLD cache build for ThinkingCap** (bfp4/bfp8 conversion); cfx-llm2 has 4 cores, so modest | Yes (host) | | nice-to-have for cold boots | | [#55465](https://github.com/tenstorrent/tt-metal/pull/55465) kernel prewarm JIT capture | 09-10 | Pre-captures JIT kernels | Boot time | Yes | | watch | | [#56832](https://github.com/tenstorrent/tt-metal/pull/56832) qwen36 reference-accuracy gate | 09-24 | `text_demo.py -k accuracy_512`, top-1/top-5 vs HF (98.63%/100% on P150x4) | A ready accuracy gate for bf8 KV / GDN kernel / #57377 numerics changes (Qwen3.6 refpt, not ThinkingCap) | Yes | | adopt the harness idea; generate a ThinkingCap refpt | | [#55548](https://github.com/tenstorrent/tt-metal/pull/55548) MTP spec decode | 09-24 | +9403/-515; new commits trace the drafter chain + reseed, shard the drafter argmax, and address review | MTP directive | Built on QB2 B=1; our tree already merges it | 2.62x @ ISL 128 | re-merge head `113099f0` into the MTP tree | | [#56939](https://github.com/tenstorrent/tt-metal/pull/56939) Llama prefill: scope cache gather to the prefix | 09-24 | Per-chunk cost follows the prompt, not the allocated cache | Pattern check: does our chunked SDPA read `max_model_len`-sized ranges? The sweep holds `max_model_len`=34816 fixed | pattern | 2.6x -> 1.0x capacity penalty (Galaxy) | audit only | | [#56184](https://github.com/tenstorrent/tt-metal/pull/56184) device RNG for trace replay; [#55892](https://github.com/tenstorrent/tt-metal/pull/55892) on-device grammar sampling | 09-23/16 | Non-greedy sampling on device | Only if we leave greedy | | | later | | [#53587](https://github.com/tenstorrent/tt-metal/pull/53587) / [#53482](https://github.com/tenstorrent/tt-metal/pull/53482) fused T=1 GDN decode | 09-02 (stale) | | Already grafted into prod | | | nothing to do | | [#56119](https://github.com/tenstorrent/tt-metal/pull/56119) Qwen3.8-27B QB2 opt (draft) | 09-10 (stale) | | Already audited: refused for TP=2 | | | nothing to do | ### vllm-tt-plugin (our pin `c127c17d`, 2026-08-14) - **Merged since the pin.** [#119](https://github.com/tenstorrent/vllm-tt-plugin/pull/119) interleaves decode steps between prefill chunks. It is default ON (2 prefill steps : 1 decode step). THIRD-PARTY: worst inter-token gap 11152 -> 899 ms, +1.0% median TTFT, measured on T3K. **For our batch-makespan metric it is roughly zero-sum.** A TT step is prefill-only or decode-only, so total work does not shrink, and the last agent's TTFT gets slightly worse. It helps interactivity, not makespan, so do not count it as a win on sweep-32k. Also merged: - [#126](https://github.com/tenstorrent/vllm-tt-plugin/pull/126) reverts the decode-first warmup. Prod is already immune. - [#79](https://github.com/tenstorrent/vllm-tt-plugin/pull/79) gates chunked prefill on a capability. - [#78](https://github.com/tenstorrent/vllm-tt-plugin/pull/78) async decode reload planning. - [#118](https://github.com/tenstorrent/vllm-tt-plugin/pull/118) adaptive spec block-output. - [#120](https://github.com/tenstorrent/vllm-tt-plugin/pull/120) / [#128](https://github.com/tenstorrent/vllm-tt-plugin/pull/128) spec contract and host accept walk. - [#135](https://github.com/tenstorrent/vllm-tt-plugin/pull/135) scheduler cleanup across prefill/decode fallback. - **Open.** - Spec decode chain: [#125](https://github.com/tenstorrent/vllm-tt-plugin/pull/125) config admit, [#127](https://github.com/tenstorrent/vllm-tt-plugin/pull/127) `[B,1+K]` block, [#130](https://github.com/tenstorrent/vllm-tt-plugin/pull/130) verify-then-propose, [#131](https://github.com/tenstorrent/vllm-tt-plugin/pull/131) on-device drafter. This is the vLLM side ThinkingCap MTP needs. - [#114](https://github.com/tenstorrent/vllm-tt-plugin/pull/114) host fallback for greedy-only device samplers overlaps our 0001. - [#22](https://github.com/tenstorrent/vllm-tt-plugin/pull/22) MambaSpec state ownership (stale draft) overlaps our 0002-0005 prefix work. - [#101](https://github.com/tenstorrent/vllm-tt-plugin/pull/101) vLLM 0.28 (draft). - **Adopt.** A plugin bump is now possible on the local tree (#55343 is present). On prod it needs the metal bump first. Our 6 plugin patches plus the bind-mounted `model_runner.py` will conflict with #118/#119/#135 in `scheduler.py`/`model_runner.py`. ### tenstorrent/vllm and tt-inference-server - **tenstorrent/vllm.** The default branch `main` was last pushed 2026-09-16. I found no `dev` branch and no Qwen/TT activity. The plugin is the live integration point, so there is nothing to take here. - **tt-inference-server** (our pin is 08-12). - Qwen3.6-27B appears only as a P300X2 `impl=quetzal` row ([#5147](https://github.com/tenstorrent/tt-inference-server/pull/5147); a generated implementation with max_context 8192 and concurrency 1) plus the existing `qwen36_blackhole` row. - Gemma4 MTP/dFlash spec serving specs ([#5137](https://github.com/tenstorrent/tt-inference-server/pull/5137)) and `--spec-decode-metrics-url` ([#4978](https://github.com/tenstorrent/tt-inference-server/pull/4978)) are useful when we instrument MTP acceptance. - P150X4 was dropped from nightly CI ([#5220](https://github.com/tenstorrent/tt-inference-server/pull/5220)). That means **less upstream CI coverage for P150 multi-card**, not more. - No performance items. ## Recommended action **A selective path, not a wholesale pin bump.** The production image is on a 09-02 base with grafted open PRs, and main has since landed a qwen36 MoE refactor, Metal 2.0 ports of the matmul/SDPA/paged-cache factories, and a P150-breaking RS heuristic change. A bump means rebasing all 23 metal patches plus 5 prod-local files, and it moves ttsim with it. Order: 1. **Now, Python only, no rebuild.** Port #57377 into `patches/prod-local-metal/.../gdn/tp.py` + `tp_common.py`, behind `QWEN36_GDN_OUT_COLPAR` (default on for max_model_len <= 34816, off for the parked long shapes). Build the `out.colpar` cache once. Also port #56471 (warm boot). Audit #54565 and #55689 against our generator and 0018. 2. **Probe before building.** Run a single-op microbenchmark of our down-proj / gate-up / QKV DRAM-sharded matmuls at M=32 with `in0_block_w` in {1, 2, 4, 8} on a #55444 build. Run the `num_workers_per_dram_bank=2` A/B on the current image (config only). Promote #55444 only if the op time moves. 3. **Next image rebuild (on nixos, not cfx-llm2)**, cherry-picked onto the prod fork: #55444, #53790 (+ #57368 tests), #53627, #53618. Resolve against 0004 and the grafted #53587. Carry #57507 pre-emptively. 4. **Wait for merge**: #57440/#57445 and #57395. Then either do a second cherry-pick round or, if the list keeps growing, do one coordinated bump (tt-metal + ttsim together) to a main commit that is **after #57507 merges** and outside the #48280 and #53557 windows. Re-merge #55548 head in that bump. 5. **MTP track (parallel).** Re-merge #55548 @ `113099f0`. Use #53430 for variable-row KV commit, the #56048 adaptive block-output pattern, and the plugin spec contract (#118/#128, then #125-#131). **Validation plan, for every adopted item, in this order:** 1. **Op/layer PCC gates.** `test_gdn_tp.py` (#57377 adds `test_gdn_out_agmm_vs_mmrs` + skew determinism), `test_chunk_gated_delta_rule.py` with the new HV=48/TP=2 geometry (#57368), and the SDPA prefill/decode unit tests at our shape (12q/2kv, d256, bf8 KV). 2. **Fixed-width greedy equivalence** at B=4 and B=8 against the current reference (`bench/runs/batch-equiv-ref-*`, bucketing off). #57377 and #57445 are **not** bit-exact: they are PCC-class numerics changes, so expect the hash to move. Use the accuracy gate (Gate A `bench/kv-accuracy.py` at 16k/32k plus GSM8K / infbench stage 3) instead of demanding sha identity. #53790, #53627 and fused-vs-phased are claimed bit-exact, so require sha identity for those. 3. **Arm self-identification.** Add a banner per new lever (like the 0016 `QWEN_SDPA_BF8` banner) so every record proves its arm. 4. **Performance.** `bench/ab/sweep-32k.sh`, 4k/8k/16k/32k x B={4,8}, current optimized arm vs arm + lever, one source tree differing only by env, `max_model_len` 34816. **Rank by makespan**, and report TTFT and decode ms/token alongside. Report PP tok/s and TG ms/token in every status (owner rule). For decode-side items (#55444, #53627), the B=4 control decode must stay faster than the B=8 baseline (existing tell). 5. **Cache arm.** Re-run `bench/ab/cache-speed-16k-32k.sh` for prefill-side items, because prefix caching changes which prefill tokens are actually computed. 6. **Record everything** in `bench/runs/*.jsonl` with the upstream PR number and commit in the config. ## Risks - **#57053 P150 FATAL.** A bump past 09-22 without #57507 cannot run qwen36 prefill on P150 at chunk >= 288 tokens. - **#57377 memory.** It is on by default with no opt-out. It adds ~+764 MiB/device weights at TP=2 (partly offset by dropped fp32 RS trace buffers, -97 MiB at TP=4) and cuts ~21% of the 3.6 GiB margin at 8x128k. This is fine at the <=32k benchmark. Gate it ourselves. - **Numerics changes that break sha-equivalence by design:** #57377 (bf16 gate output before matmul), #57445 (SFPU T_inv), #57180 (accurate exp now really accurate), #56225 (SDPA decode correction). Decide per item which bar applies before the run, not after. - **Rebase load.** #53917 (qwen36 MoE refactor) and the Metal 2.0 factory ports (matmul DRAM-sharded #55961/#56114/#56761/#56836, sdpa #54468, sdpa_decode #54249, paged_cache #56022) collide with 0004, 0020, 0023 and the grafted #53587/#53314/#53319. - **Coverage mismatch.** Almost all upstream qwen36 numbers are TP=4 (P150x4 `bh_quietbox_2`, or 4x p300c QB2). Only #53587 (stale) targets 1x2. TP=2 has twice the heads per chip (BH=24), so every kernel cost model (fused GDN geometry, SDPA chain gates) runs outside its calibrated points. - **Pinning windows to avoid:** 09-21..09-23 (fabric express routing, reverted), 09-12..09-17 (stable-topk non-finite logprobs), and before #57517 if `rms_allgather` is ever used. - **UMD bump (#57252).** It may require a matching tt-kmd / firmware. Firmware updates are never unattended. - **Shallow clone.** Do not use local `git merge-base` on `/home/orchestra/ttm-build/src` for "is X in our tree" questions; use the GitHub compare API (see skill `in-tree.sh`). ## Sources - Our repo: `flake.nix`/`flake.lock`, `patches/{tt-metal,prod-local-metal,prod-local-plugin,vllm-tt-plugin,upstream}/`, `patches/PROD-SERVING-SOURCES.md`, `docs/UPSTREAM-WATCH-PLUGIN-METAL.md`, `docs/CXX-BUILD.md`, `docker/Dockerfile.vllm-tt-k2*`, `docs/DIRECTIVES-LEDGER.md`, `bench/runs/n1n2-20260912/n2-raw.jsonl`, `docs/TRACY-PREFILL-PROFILE.md`. - GitHub API (gh, authenticated): `repos/tenstorrent/tt-metal/commits?sha=main&since=2026-08-28` (1,104 commits, 326 after keyword filter), path-scoped commit lists for `models/demos/blackhole/qwen36`, `…/transformer/sdpa`, `…/operations/experimental`; open-PR keyword search (228 PRs); `compare/<c>...<base>` for ancestry; PR bodies for every item above. - vllm-tt-plugin commits since 2026-09-01 and its open PRs; tt-inference-server commits since 2026-08-20; tenstorrent/vllm repo metadata.
Grok commented 2026-09-24 20:04:20 +02:00
Author
Owner
Copy link

agy research (2026-09-24)

Findings & Memory Accounting (#64 item 1, #41):
• PR #57377 ported: GDN prefill out-proj switched to column-parallel via all_gather_minimal_matmul_async behind QWEN36_GDN_OUTPROJ_COLPAR=1 (default 0/OFF).
• Memory cost: Shards [6144, 2560] bfp8_b per device across 48 layers = 764.99 MiB/device duplicate weights (decode retains row-sharded copy) + 50.33 MiB shared DRAM ping-pong buffer on tt_ccl.
• KV Budget Impact: At the standard <=32k benchmark (KV cache <1.1 GiB/device), +765 MiB is safe. On parked 128k/256k target shapes (16 GiB KV + 12.55 GiB weights + 1 GiB trace = ~29.5 / 30.8 GiB usable), +765 MiB consumes ~59% of the remaining ~1.3 GiB margin and competes with the prefix cache pool (1,017,275 tokens, prefix-traced-dram-oom-20260918T1540Z.jsonl).
• Local Alternative vs PR #57377: On our 1x2 P150a mesh, _fuse_out_mmrs_prefill is ALREADY FALSE due to Patch 0007 deadlock guard (args.num_devices == 2 and 1 in _mesh_shape). Current baseline runs _row_proj (computing full 5120 cols locally, 2x matmul FLOPs of col-par) + tt_all_reduce. Column-parallel (PR #57377) directly yields fractured [1, T, 2560] residual tokens with zero post-projection CCL.
• Recommendation: A/B PR #57377 (QWEN36_GDN_OUTPROJ_COLPAR=1) FIRST on <=32k sweep where DRAM is ample; keep the zero-DRAM local reshape as fallback if 128k ladder hits DRAM exhaustion.

What Changed (branch agy/issue-64, commit e288998):
• patches/prod-local-metal/models/demos/blackhole/qwen36/tt/gdn/tp.py: gated tw["out_colpar"]\ transfer and AGMM out-proj behind QWEN36_GDN_OUTPROJ_COLPAR=1; added self-identifying banner QWEN36_GDN_OUTPROJ_COLPAR engaged. • patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py: added agmm_gather_bufferping-pong buffer and wiredpersistent_output_buffer. • modules/tenstorrent-serving.nix: added gdnOutprojColparoption, assertion, and bind-mount forgdn/tp.py. • bench/ab/sweep-32k.sh: added gdn-colpararm with fail-closed banner verification; auto-mountsgdn/tp.py. • patches/PROD-SERVING-SOURCES.md`: updated vendored files documentation.

Hardware Measurement Next:
• Command: PATCHED_DIR=patches/prod-local-metal/models/demos/blackhole/qwen36/tt ARMS="optimized gdn-colpar" OWNER=sweep32k bash bench/ab/sweep-32k.sh
• Expected Result: THIRD-PARTY (PR #57377, TP=4 p300c): TTFT -9.0% at 32k, -10% at 16k. ESTIMATE for TP=2: prefill -4..-9%, makespan -3..-7% at B=8/32k.
• Go/No-go: Makespan & TTFT win vs optimized at B=4,8 across 16k/32k, greedy output equivalence preserved, DRAM headroom confirmed safe.

Open Questions:

  1. Does activation quantization to bf16 before AGMM out-proj affect greedy output tokens vs fp32 MMRS/unfused baseline?
  2. Does the +765 MiB footprint induce DRAM OOM when prefix caching is enabled with target pool sizes?
    Cross-ref: #41
**agy research (2026-09-24)** **Findings & Memory Accounting (#64 item 1, #41):** • **PR #57377 ported**: GDN prefill out-proj switched to column-parallel via `all_gather_minimal_matmul_async` behind `QWEN36_GDN_OUTPROJ_COLPAR=1` (default 0/OFF). • **Memory cost**: Shards [6144, 2560] bfp8_b per device across 48 layers = **764.99 MiB/device** duplicate weights (decode retains row-sharded copy) + 50.33 MiB shared DRAM ping-pong buffer on `tt_ccl`. • **KV Budget Impact**: At the standard <=32k benchmark (KV cache <1.1 GiB/device), +765 MiB is safe. On parked 128k/256k target shapes (16 GiB KV + 12.55 GiB weights + 1 GiB trace = ~29.5 / 30.8 GiB usable), +765 MiB consumes ~59% of the remaining ~1.3 GiB margin and competes with the prefix cache pool (1,017,275 tokens, `prefix-traced-dram-oom-20260918T1540Z.jsonl`). • **Local Alternative vs PR #57377**: On our 1x2 P150a mesh, `_fuse_out_mmrs_prefill` is ALREADY FALSE due to Patch 0007 deadlock guard (`args.num_devices == 2 and 1 in _mesh_shape`). Current baseline runs `_row_proj` (computing full 5120 cols locally, 2x matmul FLOPs of col-par) + `tt_all_reduce`. Column-parallel (PR #57377) directly yields fractured [1, T, 2560] residual tokens with zero post-projection CCL. • **Recommendation**: A/B PR #57377 (`QWEN36_GDN_OUTPROJ_COLPAR=1`) FIRST on <=32k sweep where DRAM is ample; keep the zero-DRAM local reshape as fallback if 128k ladder hits DRAM exhaustion. **What Changed (branch `agy/issue-64`, commit `e288998`):** • `patches/prod-local-metal/models/demos/blackhole/qwen36/tt/gdn/tp.py`: gated `tw["out_colpar"]\ transfer and AGMM out-proj behind `QWEN36_GDN_OUTPROJ_COLPAR=1`; added self-identifying banner `QWEN36_GDN_OUTPROJ_COLPAR engaged`. • `patches/prod-local-metal/models/demos/blackhole/qwen36/tt/tp_common.py`: added `agmm_gather_buffer` ping-pong buffer and wired `persistent_output_buffer`. • `modules/tenstorrent-serving.nix`: added `gdnOutprojColpar` option, assertion, and bind-mount for `gdn/tp.py`. • `bench/ab/sweep-32k.sh`: added `gdn-colpar` arm with fail-closed banner verification; auto-mounts `gdn/tp.py`. • `patches/PROD-SERVING-SOURCES.md`: updated vendored files documentation. **Hardware Measurement Next:** • **Command**: `PATCHED_DIR=patches/prod-local-metal/models/demos/blackhole/qwen36/tt ARMS="optimized gdn-colpar" OWNER=sweep32k bash bench/ab/sweep-32k.sh` • **Expected Result**: THIRD-PARTY (PR #57377, TP=4 p300c): TTFT -9.0% at 32k, -10% at 16k. ESTIMATE for TP=2: prefill -4..-9%, makespan -3..-7% at B=8/32k. • **Go/No-go**: Makespan & TTFT win vs `optimized` at B=4,8 across 16k/32k, greedy output equivalence preserved, DRAM headroom confirmed safe. **Open Questions:** 1. Does activation quantization to bf16 before AGMM out-proj affect greedy output tokens vs fp32 MMRS/unfused baseline? 2. Does the +765 MiB footprint induce DRAM OOM when prefix caching is enabled with target pool sizes? Cross-ref: #41
Grok commented 2026-09-25 09:26:10 +02:00
Author
Owner
Copy link

agy continuation (2026-09-25)

1. Simulation Verification (2-Chip Mesh, TP=2)

  • Correctness probe bench/probes/test_gdn_colpar_sim.py validated on 2-chip simulated Blackhole (scripts/sim-run.sh --chips 2) at Qwen3.8-27B GDN shapes (dim=5120, value_dim=6144).
  • MEASURED (bench/runs/gdn-colpar-sim-20260925T072336Z.jsonl):
    • seq 128: COLPAR=1 vs COLPAR=0 PCC = 0.999996, max abs diff = 0.031250 (PyTorch ref PCC = 0.999967).
    • seq 2048: COLPAR=1 vs COLPAR=0 PCC = 0.999996, max abs diff = 0.031250 (PyTorch ref PCC = 0.999967).
  • Both arms produce identically fractured [1, 1, T, 2560] per-device tensors that assemble cleanly to [1, 1, T, 5120].

2. DRAM & L1 Accounting for B=8, 32k Prefill

  • DRAM Footprint:
    • Out-proj weights: tw["out_colpar"]: adds +765.0 MiB/device (bfloat8_b across 48 layers; duplicated because decode retains row-sharded weights).
    • AGMM ping-pong buffer: tpc.agmm_gather_buffer reserves 2 x [1, 1, 2048, 6144] bfloat16 = +48.0 MiB/device in DRAM.
    • Total persistent DRAM added: +813.0 MiB/device.
  • Workload Headroom:
    • <=32k benchmark (B <= 8, max_len=34816, KV cache <1.1 GiB): completely safe, >8 GiB free margin.
    • Parked 8x128k target (16 GiB KV cache): +813 MiB consumes ~63% of the ~1.3 GiB margin, directly squeezing prefix cache pool capacity.
  • L1 / Trace Budget:
    • l1_small_size=24576: Semaphores allocate on CCL reserved cores; AGMM streams tile blocks through standard Tensix CBs. Zero collisions observed or expected.
    • Prefill chunks run untraced; decode trace buffers are unaffected.

3. Changes Pushed

  • Branch agy/issue-64, commit 65971a9:
    • patches/prod-local-metal/.../gdn/tp.py: guarded import for delta rule decode packed op.
    • flake.nix & scripts/sim-run.sh: unblocked local runtime (libexpat symbol fix & writable TT_METAL_HOME cache).
    • bench/probes/test_gdn_colpar_sim.py & bench/runs/gdn-colpar-sim-20260925T072336Z.jsonl: 2-chip simulation test & evidence record.

4. Tightened Hardware Runbook & Go/No-Go

  • Command: PATCHED_DIR=patches/prod-local-metal/models/demos/blackhole/qwen36/tt ARMS="optimized gdn-colpar" OWNER=sweep32k bash bench/ab/sweep-32k.sh
  • Arm Identification: Verify server log contains QWEN36_GDN_OUTPROJ_COLPAR engaged (fail-closed check built into sweep script).
  • Go Criteria: Makespan & TTFT reduction at B=4 and B=8 across <=32k shapes with no greedy token drift.
  • No-Go Criteria: Any degradation in batch makespan/TTFT, or DRAM allocation failure when prefix caching is enabled.
**agy continuation (2026-09-25)** ### 1. Simulation Verification (2-Chip Mesh, TP=2) - Correctness probe `bench/probes/test_gdn_colpar_sim.py` validated on 2-chip simulated Blackhole (`scripts/sim-run.sh --chips 2`) at Qwen3.8-27B GDN shapes (`dim=5120`, `value_dim=6144`). - `MEASURED` (`bench/runs/gdn-colpar-sim-20260925T072336Z.jsonl`): - seq 128: COLPAR=1 vs COLPAR=0 PCC = 0.999996, max abs diff = 0.031250 (PyTorch ref PCC = 0.999967). - seq 2048: COLPAR=1 vs COLPAR=0 PCC = 0.999996, max abs diff = 0.031250 (PyTorch ref PCC = 0.999967). - Both arms produce identically fractured `[1, 1, T, 2560]` per-device tensors that assemble cleanly to `[1, 1, T, 5120]`. ### 2. DRAM & L1 Accounting for B=8, 32k Prefill - **DRAM Footprint**: - Out-proj weights: `tw["out_colpar"]`: adds +765.0 MiB/device (`bfloat8_b` across 48 layers; duplicated because decode retains row-sharded weights). - AGMM ping-pong buffer: `tpc.agmm_gather_buffer` reserves 2 x [1, 1, 2048, 6144] `bfloat16` = +48.0 MiB/device in DRAM. - Total persistent DRAM added: +813.0 MiB/device. - **Workload Headroom**: - <=32k benchmark (B <= 8, max_len=34816, KV cache <1.1 GiB): completely safe, >8 GiB free margin. - Parked 8x128k target (16 GiB KV cache): +813 MiB consumes ~63% of the ~1.3 GiB margin, directly squeezing prefix cache pool capacity. - **L1 / Trace Budget**: - `l1_small_size=24576`: Semaphores allocate on CCL reserved cores; AGMM streams tile blocks through standard Tensix CBs. Zero collisions observed or expected. - Prefill chunks run untraced; decode trace buffers are unaffected. ### 3. Changes Pushed - Branch `agy/issue-64`, commit `65971a9`: - `patches/prod-local-metal/.../gdn/tp.py`: guarded import for delta rule decode packed op. - `flake.nix` & `scripts/sim-run.sh`: unblocked local runtime (libexpat symbol fix & writable `TT_METAL_HOME` cache). - `bench/probes/test_gdn_colpar_sim.py` & `bench/runs/gdn-colpar-sim-20260925T072336Z.jsonl`: 2-chip simulation test & evidence record. ### 4. Tightened Hardware Runbook & Go/No-Go - **Command**: `PATCHED_DIR=patches/prod-local-metal/models/demos/blackhole/qwen36/tt ARMS="optimized gdn-colpar" OWNER=sweep32k bash bench/ab/sweep-32k.sh` - **Arm Identification**: Verify server log contains `QWEN36_GDN_OUTPROJ_COLPAR engaged` (fail-closed check built into sweep script). - **Go Criteria**: Makespan & TTFT reduction at B=4 and B=8 across <=32k shapes with no greedy token drift. - **No-Go Criteria**: Any degradation in batch makespan/TTFT, or DRAM allocation failure when prefix caching is enabled.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
mtp-thinkingcap
tt-quant
ttx
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#64
No description provided.