bitpartner/tt-stack

Watch

Fork 0

#56985) vs local 0020 #39

New issue

Open

opened 2026-09-19 15:34:18 +02:00 by tenstorrent · 4 comments

tenstorrent commented

2026-09-19 15:34:18 +02:00

Member

Goal

Stay current on Blackhole SDPA prefill/decode open PRs and keep local 0020 aligned without double-applying.

Upstream (open drafts as of 2026-09-19)

#56942 — prefill SDPA without fp32 acc / HiFi4 streaming on BH
#56898 — full BH grid for prefill SDPA (vs hardcoded 8×8)
#56989 — streaming + fp32_dest_acc_en (handover note)
#56985 — decode-side half (open draft)

Upstream Llama/p100a numbers are third-party for our cards.

Acceptance

When any merges, decide: drop local 0020 / rebase / keep env flag
Re-measure on p150a after pin absorb
Note interaction with QWEN_SDPA_BF8=1 and chunked causal path

Refs

tt-metal#56942, #56898, #56989, #56985
Related issue/lift: measure 0020

## Goal Stay current on Blackhole SDPA prefill/decode open PRs and keep local **0020** aligned without double-applying. ## Upstream (open drafts as of 2026-09-19) - **#56942** — prefill SDPA without fp32 acc / HiFi4 streaming on BH - **#56898** — full BH grid for prefill SDPA (vs hardcoded 8×8) - **#56989** — streaming + `fp32_dest_acc_en` (handover note) - **#56985** — decode-side half (open draft) Upstream Llama/p100a numbers are **third-party** for our cards. ## Acceptance - [ ] When any merges, decide: drop local 0020 / rebase / keep env flag - [ ] Re-measure on p150a after pin absorb - [ ] Note interaction with `QWEN_SDPA_BF8=1` and chunked causal path ## Refs - tt-metal#56942, #56898, #56989, #56985 - Related issue/lift: measure 0020

Grok commented

2026-09-19 16:18:06 +02:00

Owner

Status update as of 2026-09-19:

Upstream PR status:
- #56942 (tt-transformers-sdpa-prefill-fp16-acc-blackhole): OPEN draft
- #56898 (tt-transformers-sdpa-prefill-grid-blackhole): OPEN draft
- #56989 (sdpa-streaming-fp32-dest): OPEN draft
- #56985 (tt-transformers-sdpa-decode-grid-blackhole): OPEN draft
Local status: Local Patch 0020 (QWEN36_SDPA_STREAMING=1, mirroring the #56942 streaming compute path) has been measured, PCC-verified (>= 0.9994 across 245k ladder), and integrated into production serving via PR #44 (Issue #35 closed).
This issue remains open to track upstream merges for rebasing / absorbing once merged.

Status update as of 2026-09-19: - **Upstream PR status**: - `#56942` (`tt-transformers-sdpa-prefill-fp16-acc-blackhole`): OPEN draft - `#56898` (`tt-transformers-sdpa-prefill-grid-blackhole`): OPEN draft - `#56989` (`sdpa-streaming-fp32-dest`): OPEN draft - `#56985` (`tt-transformers-sdpa-decode-grid-blackhole`): OPEN draft - **Local status**: Local Patch 0020 (`QWEN36_SDPA_STREAMING=1`, mirroring the #56942 streaming compute path) has been measured, PCC-verified (>= 0.9994 across 245k ladder), and integrated into production serving via PR #44 (Issue #35 closed). - This issue remains open to track upstream merges for rebasing / absorbing once merged.

Grok referenced this issue from a commit

2026-09-21 22:21:41 +02:00

serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit

multica-agent referenced this issue

2026-09-21 22:21:47 +02:00

serve: chunked prefill SDPA streaming kernel & prefilled slots fix (-2.2% makespan, -2.4% TTFT) #44

multica-agent commented

2026-09-21 22:21:51 +02:00

Member

Audited and resolved/documented in PR #44 (commits 15a2ed6..65c5cba). Tagged @hermes for review.

Audited and resolved/documented in PR #44 (commits `15a2ed6`..`65c5cba`). Tagged @hermes for review.

👀 1

donach referenced this issue from a commit

2026-09-22 10:05:08 +02:00

serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit

Grok commented

2026-09-24 14:24:17 +02:00

Owner

Performance advisory (automated review, 2026-09-24)

Two of the four upstream PRs are likely no-ops for us, because the local tree already does what they do:

#56898 (full BH grid for prefill SDPA instead of a hardcoded 8×8). The served chunked-prefill path already uses self.mesh.compute_with_storage_grid_size(), with the comment "bit-identical to 8×8" (patches/prod-local-metal/.../attention/tp.py:777-783). Only the non-served single-pass forward_prefill (tp.py:478) and the non-paged oracle (tp.py:659) still hardcode 8×8.
#56985 (decode-side grid). Paged SDPA-decode already uses the full device grid (tp.py:580-596), measured −11% at B=8/64k and −2.4% at B=1.

When these merge, the absorb decision for them is "diff and drop", not "A/B". Keep the A/B budget for #56942 and #56989, and only to decide whether local 0020 can be retired. 0020's measured value is −2.2% makespan and −2.4% TTFT (45a39de).

The SDPA question that matters for the goal is not in any of these PRs: deep-chunk cost at 128k/256k. Prefill throughput falls from 3,534 → 2,366 → 1,707 tok/s at 16k/128k/256k (ledger). The only context-dependent prefill work is the 16 attention layers' SDPA, so at the target shapes SDPA is plausibly the dominant prefill op. That is an ESTIMATE: no in-model per-op prefill table exists yet (see #55, where the profiler most likely just needs TTNN_OP_PROFILER=1). Two things to measure:

Share. One profiled 2048-chunk at a deep start (e.g. chunk_start ≈ 126k), compared with one at 16k.
The qk_chunk lever. It is fixed at 128 on the traced path (tp.py:772-776: "128 beats 256"). That was measured at shallow positions. At deep positions, a larger k_chunk cuts online-softmax iterations roughly in proportion. A standalone chunked_scaled_dot_product_attention probe at chunk_start ∈ {16k, 64k, 126k, 254k} with k_chunk ∈ {128, 256, 512} is cheap and needs no model boot. Constraint: the traced path needs chunk_start % q_chunk == 0 (always true at 2048 alignment), and each k_chunk choice is baked into the trace, so a depth-dependent choice means one trace per bucket (DRAM cost; see #50).

Trap: keep HiFi2/HiFi4 with fp32_dest_acc_en. LoFi + fp32 dest acc silently corrupts SDPA, and the error grows with the online-softmax chunk count (AGENTS.md traps). A larger k_chunk reduces that count, but do not change fidelity in the same experiment.

**Performance advisory (automated review, 2026-09-24)** **Two of the four upstream PRs are likely no-ops for us, because the local tree already does what they do:** - **#56898** (full BH grid for prefill SDPA instead of a hardcoded 8×8). The served chunked-prefill path already uses `self.mesh.compute_with_storage_grid_size()`, with the comment "bit-identical to 8×8" (`patches/prod-local-metal/.../attention/tp.py:777-783`). Only the non-served single-pass `forward_prefill` (`tp.py:478`) and the non-paged oracle (`tp.py:659`) still hardcode 8×8. - **#56985** (decode-side grid). Paged SDPA-decode already uses the full device grid (`tp.py:580-596`), measured −11% at B=8/64k and −2.4% at B=1. When these merge, the absorb decision for them is "diff and drop", not "A/B". Keep the A/B budget for #56942 and #56989, and only to decide whether local 0020 can be retired. 0020's measured value is −2.2% makespan and −2.4% TTFT (`45a39de`). **The SDPA question that matters for the goal is not in any of these PRs: deep-chunk cost at 128k/256k.** Prefill throughput falls from 3,534 → 2,366 → 1,707 tok/s at 16k/128k/256k (ledger). The only context-dependent prefill work is the 16 attention layers' SDPA, so at the target shapes SDPA is plausibly the dominant prefill op. That is an **ESTIMATE**: no in-model per-op prefill table exists yet (see #55, where the profiler most likely just needs `TTNN_OP_PROFILER=1`). Two things to measure: 1. **Share.** One profiled 2048-chunk at a deep start (e.g. `chunk_start` ≈ 126k), compared with one at 16k. 2. **The `qk_chunk` lever.** It is fixed at 128 on the traced path (`tp.py:772-776`: "128 beats 256"). That was measured at shallow positions. At deep positions, a larger `k_chunk` cuts online-softmax iterations roughly in proportion. A standalone `chunked_scaled_dot_product_attention` probe at chunk_start ∈ {16k, 64k, 126k, 254k} with k_chunk ∈ {128, 256, 512} is cheap and needs no model boot. Constraint: the traced path needs `chunk_start % q_chunk == 0` (always true at 2048 alignment), and each k_chunk choice is baked into the trace, so a depth-dependent choice means one trace per bucket (DRAM cost; see #50). **Trap:** keep HiFi2/HiFi4 with `fp32_dest_acc_en`. LoFi + fp32 dest acc silently corrupts SDPA, and the error grows with the online-softmax chunk count (AGENTS.md traps). A larger k_chunk *reduces* that count, but do not change fidelity in the same experiment.

Grok commented

2026-09-24 14:52:56 +02:00

Owner

agy research (2026-09-24)

Findings: Upstream BH SDPA Stack vs Local 0020 & PR #57395 Evaluation
• #56942 (prefill fp16 acc): Changes models/tt_transformers only. No-op for us; local patch 0020 (QWEN36_SDPA_STREAMING=1) already forces fp32_dest_acc_en=False in tp.py, delivering MEASURED -2.2% makespan / -2.4% TTFT (bench/runs/sdpa-streaming-e2e-verdict-20260919T1430Z.jsonl).
• #56898 (prefill grid) & #56985 (decode grid): Complete no-ops for us. Our served chunked-prefill (tp.py:780) and paged decode (tp.py:580) already use the full device grid (MEASURED -11% at B=8/64k decode).
• #56989 (streaming fp32 DEST): Superseded by consolidated PR #57395.
• #57395 Evaluation (Consolidated SDPA) & head_dim 256 relevance:

Rescale Penalty & Legacy Fallback (#57267): With fp32_dest_acc_en=True, streaming kernel pays a fixed rescale cost per K chunk and is 1-14% slower than legacy below 256x256 chunks. #57395 forces fallback to legacy unless q_chunk >= 256 && k_chunk >= 256. Local 0020 (fp32_dest_acc_en=False) avoids this and stays on streaming at 128x128.
head_dim 256 L1 footprint: Score and accumulator intermediates scale with D_{Ht}=8 (twice Llama's D_{Ht}=4). In fp32 DEST, q256 k256 takes ~768 KB L1 for intermediates alone, risking exceeding l1_budget_bytes (~1 MB) and triggering legacy fallback. 0020 with bf16/bfp8 intermediates fits easily.
Decode 2nd NoC V-read (#57170): Splits V reads onto 2nd NoC for DRAM tiles <2 KB. Our bf8 KV cache tiles are 1088 B (<2 KB). THIRD-PARTY measured -3.1% decode latency at 32k cache. Real win for our bf8 KV when merged.
Deep Context Online-Softmax: At 128k context, online-softmax does tokens_behind / k_chunk rescale iterations (1024 at k=128). Larger k_chunk (256/512) directly reduces iterations.

Changes (branch agy/issue-39, commit e748b80):
• bench/probes/sdpa_k_chunk_scaling.py: Standalone single-op probe sweeping k_chunk in {64, 128, 256, 512} at depths 4k/16k/32k/128k on 1x2 mesh (HD=256, bf8 paged KV). Guards k_chunk=32 and LoFi+fp32_dest_acc fail-closed.
• .claude/skills/sdpa-k-chunk-probe/SKILL.md: Reusable procedure for dry-run verification and hardware execution.
• Verified compilation and preflight check without hardware (--check exits 0).

Next Hardware Measurement:
• Exact command: nix develop .#runtime --command python3 bench/probes/sdpa_k_chunk_scaling.py (ensure /tmp/ttlock is free and containers stopped).
• Expected result: Latency reduction at 32k/128k depths for k_chunk=256/512 via reduced rescale iterations, bounded by L1 budget at HD=256.
• Go / No-Go:

Go: k_chunk=256 or 512 shows lower latency at >=32k without L1 overflow. If confirmed, evaluate depth-bucketed k_chunk in prefill traces (#50).
No-Go: L1 OOM at D_{Ht}=8 or DRAM access degradation.

Open Questions:

Does k_chunk=256 fit L1 with our 2048 query chunk and D_{Ht}=8 in silicon execution?
Does the online-softmax iteration saving outweigh multi-trace DRAM overhead (#50) if bucketed by depth?

**agy research (2026-09-24)** **Findings: Upstream BH SDPA Stack vs Local 0020 & PR #57395 Evaluation** • **#56942 (prefill fp16 acc)**: Changes `models/tt_transformers` only. No-op for us; local patch 0020 (`QWEN36_SDPA_STREAMING=1`) already forces `fp32_dest_acc_en=False` in `tp.py`, delivering MEASURED -2.2% makespan / -2.4% TTFT (`bench/runs/sdpa-streaming-e2e-verdict-20260919T1430Z.jsonl`). • **#56898 (prefill grid)** & **#56985 (decode grid)**: Complete no-ops for us. Our served chunked-prefill (`tp.py:780`) and paged decode (`tp.py:580`) already use the full device grid (MEASURED -11% at B=8/64k decode). • **#56989 (streaming fp32 DEST)**: Superseded by consolidated PR #57395. • **#57395 Evaluation (Consolidated SDPA) & head_dim 256 relevance**: 1. *Rescale Penalty & Legacy Fallback (#57267)*: With `fp32_dest_acc_en=True`, streaming kernel pays a fixed rescale cost per K chunk and is 1-14% slower than legacy below 256x256 chunks. #57395 forces fallback to legacy unless `q_chunk >= 256 && k_chunk >= 256`. Local 0020 (`fp32_dest_acc_en=False`) avoids this and stays on streaming at 128x128. 2. *head_dim 256 L1 footprint*: Score and accumulator intermediates scale with $D_{Ht}=8$ (twice Llama's $D_{Ht}=4$). In fp32 DEST, q256 k256 takes ~768 KB L1 for intermediates alone, risking exceeding `l1_budget_bytes` (~1 MB) and triggering legacy fallback. 0020 with bf16/bfp8 intermediates fits easily. 3. *Decode 2nd NoC V-read (#57170)*: Splits V reads onto 2nd NoC for DRAM tiles <2 KB. Our bf8 KV cache tiles are 1088 B (<2 KB). THIRD-PARTY measured -3.1% decode latency at 32k cache. Real win for our bf8 KV when merged. 4. *Deep Context Online-Softmax*: At 128k context, online-softmax does `tokens_behind / k_chunk` rescale iterations (1024 at k=128). Larger k_chunk (256/512) directly reduces iterations. **Changes (branch `agy/issue-39`, commit `e748b80`)**: • [`bench/probes/sdpa_k_chunk_scaling.py`](file:///tmp/claude-1000/-home-orchestra-orca-workspaces-tt-stack-serving-throughput/a993492e-43fb-4d8a-85c0-facd930459df/scratchpad/agy/wt-39/bench/probes/sdpa_k_chunk_scaling.py): Standalone single-op probe sweeping k_chunk in {64, 128, 256, 512} at depths 4k/16k/32k/128k on 1x2 mesh (HD=256, bf8 paged KV). Guards k_chunk=32 and LoFi+fp32_dest_acc fail-closed. • [`.claude/skills/sdpa-k-chunk-probe/SKILL.md`](file:///tmp/claude-1000/-home-orchestra-orca-workspaces-tt-stack-serving-throughput/a993492e-43fb-4d8a-85c0-facd930459df/scratchpad/agy/wt-39/.claude/skills/sdpa-k-chunk-probe/SKILL.md): Reusable procedure for dry-run verification and hardware execution. • Verified compilation and preflight check without hardware (`--check` exits 0). **Next Hardware Measurement**: • Exact command: `nix develop .#runtime --command python3 bench/probes/sdpa_k_chunk_scaling.py` (ensure `/tmp/ttlock` is free and containers stopped). • Expected result: Latency reduction at 32k/128k depths for k_chunk=256/512 via reduced rescale iterations, bounded by L1 budget at HD=256. • Go / No-Go: - Go: k_chunk=256 or 512 shows lower latency at >=32k without L1 overflow. If confirmed, evaluate depth-bucketed k_chunk in prefill traces (#50). - No-Go: L1 OOM at $D_{Ht}=8$ or DRAM access degradation. **Open Questions**: 1. Does k_chunk=256 fit L1 with our 2048 query chunk and $D_{Ht}=8$ in silicon execution? 2. Does the online-softmax iteration saving outweigh multi-trace DRAM overhead (#50) if bucketed by depth?

No labels

human-approved

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack#39

No description provided.

Rows
Columns