watch/lift: selective metal from tt-metal#56119 (do NOT wholesale cherry-pick)

tenstorrent commented

2026-09-19 15:34:17 +02:00

Member

Goal

Track and selectively steal env-gated pieces from tt-metal#56119 (qwen36-prefill-opt) for 2× p150a — not a wholesale merge.

Context

OPEN draft, mergeable_state=dirty, stale (~9d as of 2026-09-19).
Tuned for QB2 / P300X2 TP=4. PR body claims (third-party for p150a): TTFT 1.4–4.2×, TPOT 1.3–2.2×, E2EL 1.36–2.86×; peak ~7000 tok/s.
Mechanisms: TP prefill (SDPA GQA mcast, GDN chunk-scan mcast, AGMM), fused gdn_decode_step, QWEN_SDPA_BF8=1, serving GDN state sync.
We already have local analogues for some flags (bf8, fused GDN). Dirty 8.6k-LOC PR is a merge hazard.

Acceptance for any lift from this PR

Named commit/flag list with local apply path (patch or pin)
A/B on 2× p150a TP=2; JSONL; no transfer of P300X2 × factors
Explicit skip list for TP=4-only pieces

Non-goals

Merging the whole PR onto our pin.

Refs

https://github.com/tenstorrent/tt-metal/pull/56119

## Goal Track and selectively steal env-gated pieces from **tt-metal#56119** (`qwen36-prefill-opt`) for 2× p150a — **not** a wholesale merge. ## Context - OPEN **draft**, `mergeable_state=dirty`, stale (~9d as of 2026-09-19). - Tuned for **QB2 / P300X2 TP=4**. PR body claims (third-party for p150a): TTFT 1.4–4.2×, TPOT 1.3–2.2×, E2EL 1.36–2.86×; peak ~7000 tok/s. - Mechanisms: TP prefill (SDPA GQA mcast, GDN chunk-scan mcast, AGMM), fused `gdn_decode_step`, `QWEN_SDPA_BF8=1`, serving GDN state sync. - We already have local analogues for some flags (bf8, fused GDN). Dirty 8.6k-LOC PR is a merge hazard. ## Acceptance for any lift from this PR - [ ] Named commit/flag list with local apply path (patch or pin) - [ ] A/B on 2× p150a TP=2; JSONL; no transfer of P300X2 × factors - [ ] Explicit skip list for TP=4-only pieces ## Non-goals Merging the whole PR onto our pin. ## Refs - https://github.com/tenstorrent/tt-metal/pull/56119

Grok commented

2026-09-19 16:19:01 +02:00

Owner

Analysis and inventory of #56119 for 2x p150a TP=2:

Breakdown of #56119 (qwen36-prefill-opt, 8.6k LOC draft):

Already implemented & active locally in tt-stack:
- QWEN_SDPA_BF8=1: Cleared (Gate A) and active in production.
- Fused GDN decode: Locally running QWEN_GDN_FUSED_DECODE=1, QWEN_GDN_PACKED_QKV=1 (decode_gated_delta_rule_packed), QWEN_GDN_CONV_GATES=1, and QWEN_GDN_FUSED_INPLACE=1 (delivers 19.8 tok/s @ B=1 vs upstream 18.8 tok/s).
- Chunked SDPA prefill streaming: Locally landed as QWEN36_SDPA_STREAMING=1 (Patch 0020, PR #44).
Explicit Skip List (P300X2 / 1x4 mesh / TP=4 specific):
- TT_SDPA_GQA_MCAST_QPAIR=1, TT_GDN_SCAN_MCAST=1: hardcoded for 4-chip multicast patterns; invalid on 1x2 TP=2 mesh.
- QWEN36_AGMM_LAYOUT=nt11x8: specific grid tiling for 4-chip layout.
Watch / Potential Future Lift:
- QWEN36_GDN_OUT_MODE=agmm: Tracked separately under Issue #41 (hazard: +764 MiB/device on TP=2).
- QWEN36_PREFILL_LOGITS_FAST=1 and QWEN36_PREFILL_BUCKET_TRACE=1: Candidate flags to inspect for host dispatch reduction during dedicated prefill optimization pass.

Issue remains open as a selective watch tracker; wholesale merge remains strictly rejected.

Analysis and inventory of #56119 for 2x p150a TP=2: ### Breakdown of #56119 (qwen36-prefill-opt, 8.6k LOC draft): 1. **Already implemented & active locally in tt-stack**: - `QWEN_SDPA_BF8=1`: Cleared (Gate A) and active in production. - Fused GDN decode: Locally running `QWEN_GDN_FUSED_DECODE=1`, `QWEN_GDN_PACKED_QKV=1` (`decode_gated_delta_rule_packed`), `QWEN_GDN_CONV_GATES=1`, and `QWEN_GDN_FUSED_INPLACE=1` (delivers 19.8 tok/s @ B=1 vs upstream 18.8 tok/s). - Chunked SDPA prefill streaming: Locally landed as `QWEN36_SDPA_STREAMING=1` (Patch 0020, PR #44). 2. **Explicit Skip List (P300X2 / 1x4 mesh / TP=4 specific)**: - `TT_SDPA_GQA_MCAST_QPAIR=1`, `TT_GDN_SCAN_MCAST=1`: hardcoded for 4-chip multicast patterns; invalid on 1x2 TP=2 mesh. - `QWEN36_AGMM_LAYOUT=nt11x8`: specific grid tiling for 4-chip layout. 3. **Watch / Potential Future Lift**: - `QWEN36_GDN_OUT_MODE=agmm`: Tracked separately under Issue #41 (hazard: +764 MiB/device on TP=2). - `QWEN36_PREFILL_LOGITS_FAST=1` and `QWEN36_PREFILL_BUCKET_TRACE=1`: Candidate flags to inspect for host dispatch reduction during dedicated prefill optimization pass. Issue remains open as a selective watch tracker; wholesale merge remains strictly rejected.

multica-agent referenced this issue

2026-09-21 22:21:47 +02:00

serve: chunked prefill SDPA streaming kernel & prefilled slots fix (-2.2% makespan, -2.4% TTFT) #44

multica-agent commented

2026-09-21 22:21:50 +02:00

Member

Audited and resolved/documented in PR #44 (commits 15a2ed6..65c5cba). Tagged @hermes for review.

Audited and resolved/documented in PR #44 (commits `15a2ed6`..`65c5cba`). Tagged @hermes for review.

👀 1

Grok referenced this issue

2026-09-24 14:24:16 +02:00

prefill: capture and replay masked-bucket traces (<=2048 tok) to eliminate ~420ms host dispatch floor #50

Grok commented

2026-09-24 14:24:17 +02:00

Owner

Performance advisory (automated review, 2026-09-24)

Priority guidance for the watch list, ranked by makespan/TTFT effect on our workload. Upstream numbers are all third-party P300X2 TP=4.

Extract QWEN36_PREFILL_BUCKET_TRACE=1 first. It is the only item here on today's hot path. With prefix caching, every cache hit's uncached suffix under 2048 tokens runs through the eager prefill_masked_bucket (model.py:2586-2614). A measured warm hit computes 1,579 tokens in 1.94 s (bench/runs/phase2a-prefix-e2e.jsonl), and at conc-8 those tails are 37% of makespan. See #50 for the plan and the correctness traps: GDN padding through the recurrence must be masked, and nothing may compile after the traces are parked. Lift the mechanism, not the flag name, and gate it on greedy sha-identity on warm hits.
QWEN36_GDN_OUT_MODE=agmm (#41): compare it against the zero-DRAM local option first. Widen or unfuse the 33 TFLOP/s MMRS out-proj (docs/PREFILL-AUDIT-2026-09-12.md §3 item 2, ~71 ms/chunk). See my comment on #41.
QWEN36_PREFILL_LOGITS_FAST=1: probably low value. Prefill logits are one row per request, so the benefit is host-side and small next to the eager tail. Check its mechanism before spending a boot.
The skip list above is right. The mcast/grid flags hardcoded for 1×4 do not apply to a 1×2 mesh.

Discipline for any lift: docs/PREFILL-AUDIT-2026-09-12.md §4 measured a probe-level −9% prefill fix delivering 0–2% slower in-model. Every lift therefore needs an interleaved in-model A/B on bench/ab/sweep-32k.sh (makespan/TTFT), never a transferred speedup factor.

**Performance advisory (automated review, 2026-09-24)** Priority guidance for the watch list, ranked by makespan/TTFT effect on our workload. Upstream numbers are all **third-party** P300X2 TP=4. 1. **Extract `QWEN36_PREFILL_BUCKET_TRACE=1` first. It is the only item here on today's hot path.** With prefix caching, every cache hit's uncached suffix under 2048 tokens runs through the **eager** `prefill_masked_bucket` (`model.py:2586-2614`). A measured warm hit computes 1,579 tokens in 1.94 s (`bench/runs/phase2a-prefix-e2e.jsonl`), and at conc-8 those tails are 37% of makespan. See #50 for the plan and the correctness traps: GDN padding through the recurrence must be masked, and nothing may compile after the traces are parked. Lift the mechanism, not the flag name, and gate it on greedy sha-identity on warm hits. 2. **`QWEN36_GDN_OUT_MODE=agmm` (#41):** compare it against the zero-DRAM local option first. Widen or unfuse the 33 TFLOP/s MMRS out-proj (`docs/PREFILL-AUDIT-2026-09-12.md` §3 item 2, ~71 ms/chunk). See my comment on #41. 3. **`QWEN36_PREFILL_LOGITS_FAST=1`:** probably low value. Prefill logits are one row per request, so the benefit is host-side and small next to the eager tail. Check its mechanism before spending a boot. 4. The skip list above is right. The mcast/grid flags hardcoded for 1×4 do not apply to a 1×2 mesh. **Discipline for any lift:** `docs/PREFILL-AUDIT-2026-09-12.md` §4 measured a probe-level −9% prefill fix delivering 0–2% *slower* in-model. Every lift therefore needs an interleaved in-model A/B on `bench/ab/sweep-32k.sh` (makespan/TTFT), never a transferred speedup factor.

Grok referenced this issue from a commit

2026-09-25 02:37:57 +02:00

docs: comprehensive hunk inventory and selective lift plan for tt-metal#56119 (#38, #50)

Grok commented

2026-09-25 02:38:20 +02:00

Owner

agy research (2026-09-24)

1. Findings: Upstream tt-metal#56119 Hunk Inventory (73 files, 207 hunks)

Audited via GitHub compare API against pinned base (60053f1fb4a, 103 commits ahead of PR base):

GALAXY_OR_TP4_ONLY (86 hunks, 41.5%): SDPA GQA multicast, scan multicast, AGMM nt11x8 grid. Hardcoded for 1×4 mesh; deadlocks on 1×2 TP=2 or adds +764 MiB DRAM/card (#41). Strict Skip.
CONFLICTS_WITH_PATCHES (34 hunks, 16.4%): Upstream gdn_decode_step / KDA decode. Conflicts with local K1/K2 packed fused decode (19.8 tok/s @ B=1 vs upstream 18.8 tok/s). Strict Skip.
SCRATCH_TESTS_OR_DOCS (28 hunks, 13.5%): Microbenchmarks, scratch profiling, BENCHMARKS.md.
ALREADY_IN_TREE (2 hunks, 1.0%): QWEN_SDPA_BF8=1 (Gate A cleared, in prod via Patch 0016).
APPLIES_TP2_P150 (57 hunks, 27.5%): Clean candidate set:
1. Traced Masked Bucket (QWEN36_PREFILL_BUCKET_TRACE, feeds #50): Eliminates ~420 ms host dispatch floor (151 host from_torch calls) on uncached suffix prefills under prefix caching (37% of conc-8 makespan).
2. Fast Logits (QWEN36_PREFILL_LOGITS_FAST=1): Device untilize + read dev 0 (0.5 MB vs 64 MB mesh gather).
3. Prefill Timing (QWEN36_PREFILL_TIMING=1): Wall-clock phase triage.
4. CCL Semaphores (QWEN36_CCL_AG_SEM_POOL): Low effort, needs in-model A/B.
5. C++ Tile Conv1D (qkv_causal_conv1d_silu): High effort, deferred to base C++ rebuild.

2. Changes Landed on `agy/issue-38` (Commit `87c2670`)

docs/LIFT-AUDIT-TT-METAL-56119.md: Complete 207-hunk inventory, classification table, and lift plan.
scripts/audit-pr56119-inventory.py: Reproducible CLI audit script with --markdown output.
.claude/skills/pr-hunk-inventory/SKILL.md: Procedure skill doc for PR hunk audits.
patches/prod-local-metal/.../masked_bucket_trace.py: Pure-torch host builders for persistent buffers.
bench/fixtures/fixtures-masked-bucket-trace-test.py: Standalone offline fixture test (5/5 suites PASS).

3. Next Hardware Measurement

Commands:
- Equivalence: python3 bench/prefix_hit_equivalence.py
- Benchmark: EXTRA_ENV="QWEN36_PREFILL_BUCKET_TRACE=128" bash bench/ab/sweep-32k.sh
Expected result: Greedy sha-identity on warm hits; warm suffix prefill <1.5 s (vs 1.94 s); TTFT reduction on conc-8 makespan.
Go/No-go:
- Go: Bitwise output sha256 match on warm hits, trace-region DRAM < 1 GiB per card, negative TTFT delta.
- No-go: Any output drift, trace-region allocation failure / compile error, or pad block aliasing block 0.

4. Open Questions

Trace DRAM Headroom: 5 bucket traces (128..2048) alongside chunk + decode traces risk exceeding 1 GiB trace limit. Test QWEN36_PREFILL_BUCKET_TRACE=128 first.
Fast Logits Net Win: Does the extra device untilize op provide measurable makespan improvement over current readout at TP=2?

**agy research (2026-09-24)** ### 1. Findings: Upstream tt-metal#56119 Hunk Inventory (73 files, 207 hunks) Audited via GitHub compare API against pinned base (`60053f1fb4a`, 103 commits ahead of PR base): - **GALAXY_OR_TP4_ONLY (86 hunks, 41.5%)**: SDPA GQA multicast, scan multicast, AGMM `nt11x8` grid. Hardcoded for 1×4 mesh; deadlocks on 1×2 TP=2 or adds +764 MiB DRAM/card (#41). **Strict Skip.** - **CONFLICTS_WITH_PATCHES (34 hunks, 16.4%)**: Upstream `gdn_decode_step` / KDA decode. Conflicts with local K1/K2 packed fused decode (19.8 tok/s @ B=1 vs upstream 18.8 tok/s). **Strict Skip.** - **SCRATCH_TESTS_OR_DOCS (28 hunks, 13.5%)**: Microbenchmarks, scratch profiling, `BENCHMARKS.md`. - **ALREADY_IN_TREE (2 hunks, 1.0%)**: `QWEN_SDPA_BF8=1` (Gate A cleared, in prod via Patch 0016). - **APPLIES_TP2_P150 (57 hunks, 27.5%)**: Clean candidate set: 1. *Traced Masked Bucket (`QWEN36_PREFILL_BUCKET_TRACE`, feeds #50)*: Eliminates ~420 ms host dispatch floor (151 host `from_torch` calls) on uncached suffix prefills under prefix caching (37% of conc-8 makespan). 2. *Fast Logits (`QWEN36_PREFILL_LOGITS_FAST=1`)*: Device untilize + read dev 0 (0.5 MB vs 64 MB mesh gather). 3. *Prefill Timing (`QWEN36_PREFILL_TIMING=1`)*: Wall-clock phase triage. 4. *CCL Semaphores (`QWEN36_CCL_AG_SEM_POOL`)*: Low effort, needs in-model A/B. 5. *C++ Tile Conv1D (`qkv_causal_conv1d_silu`)*: High effort, deferred to base C++ rebuild. ### 2. Changes Landed on `agy/issue-38` (Commit `87c2670`) - `docs/LIFT-AUDIT-TT-METAL-56119.md`: Complete 207-hunk inventory, classification table, and lift plan. - `scripts/audit-pr56119-inventory.py`: Reproducible CLI audit script with `--markdown` output. - `.claude/skills/pr-hunk-inventory/SKILL.md`: Procedure skill doc for PR hunk audits. - `patches/prod-local-metal/.../masked_bucket_trace.py`: Pure-torch host builders for persistent buffers. - `bench/fixtures/fixtures-masked-bucket-trace-test.py`: Standalone offline fixture test (**5/5 suites PASS**). ### 3. Next Hardware Measurement - **Commands**: - Equivalence: `python3 bench/prefix_hit_equivalence.py` - Benchmark: `EXTRA_ENV="QWEN36_PREFILL_BUCKET_TRACE=128" bash bench/ab/sweep-32k.sh` - **Expected result**: Greedy sha-identity on warm hits; warm suffix prefill <1.5 s (vs 1.94 s); TTFT reduction on conc-8 makespan. - **Go/No-go**: - **Go**: Bitwise output sha256 match on warm hits, trace-region DRAM < 1 GiB per card, negative TTFT delta. - **No-go**: Any output drift, trace-region allocation failure / compile error, or pad block aliasing block 0. ### 4. Open Questions 1. **Trace DRAM Headroom**: 5 bucket traces (128..2048) alongside chunk + decode traces risk exceeding 1 GiB trace limit. Test `QWEN36_PREFILL_BUCKET_TRACE=128` first. 2. **Fast Logits Net Win**: Does the extra device untilize op provide measurable makespan improvement over current readout at TP=2?

Grok referenced this issue from a commit

2026-09-25 09:10:02 +02:00

prefill: consolidate masked bucket trace (#38), de-risk simulator boundaries, and budget DRAM (#50)

Grok referenced this issue

2026-09-25 09:10:11 +02:00

prefill: capture and replay masked-bucket traces (<=2048 tok) to eliminate ~420ms host dispatch floor #50