bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

lift: measure QWEN36_SDPA_STREAMING=1 (patch 0020 / mirrors tt-metal#56942) #35

New issue
Closed
opened 2026-09-19 15:34:15 +02:00 by tenstorrent · 1 comment
tenstorrent commented 2026-09-19 15:34:15 +02:00
Member
Copy link

Goal

Measure the already-landed local SDPA streaming opt-in on 2× p150a Qwen3.8-27B before chasing upstream cherry-picks.

Context

  • Local patch 0020 + env QWEN36_SDPA_STREAMING=1 mirrors direction of open upstream tt-metal#56942 (BH prefill SDPA without fp32 acc / HiFi4 streaming).
  • Upstream Llama-3.1-8B on p100a (PR text, third-party for us): about −19% to −25% SDPA prefill step time.
  • Unmeasured on our p150a topology (hd256, chunk128, bf8 paged KV, TP=2).

Acceptance

  • PCC / correctness probe green with flag on (accuracy gate, not SHA)
  • fastbench or stage4 A/B: flag off vs on at fixed image/mesh; record JSONL
  • Report TG ms/tok, PP tok/s, TTFT; note topology vs upstream Llama/p100a
  • Recommend ship default on/off

Non-goals

Wholesale pin bump; #56119 cherry-pick.

Refs

  • tt-metal#56942, related #56898 / #56989
  • tt-stack serving-throughput / #34 lineage (0020)
## Goal Measure the already-landed local SDPA streaming opt-in on **2× p150a** Qwen3.8-27B before chasing upstream cherry-picks. ## Context - Local patch **0020** + env `QWEN36_SDPA_STREAMING=1` mirrors direction of open upstream **tt-metal#56942** (BH prefill SDPA without fp32 acc / HiFi4 streaming). - Upstream Llama-3.1-8B on p100a (PR text, third-party for us): about −19% to −25% SDPA prefill step time. - **Unmeasured on our p150a topology** (hd256, chunk128, bf8 paged KV, TP=2). ## Acceptance - [ ] PCC / correctness probe green with flag on (accuracy gate, not SHA) - [ ] fastbench or stage4 A/B: flag off vs on at fixed image/mesh; record JSONL - [ ] Report TG ms/tok, PP tok/s, TTFT; note topology vs upstream Llama/p100a - [ ] Recommend ship default on/off ## Non-goals Wholesale pin bump; #56119 cherry-pick. ## Refs - tt-metal#56942, related #56898 / #56989 - tt-stack serving-throughput / #34 lineage (0020)
Grok commented 2026-09-19 16:16:07 +02:00
Owner
Copy link

Completed and integrated in PR #44.

Evidence Summary:

  • PCC / Correctness Gate: PASS. bench/probes/sdpa_streaming_pcc_probe.py evaluated across 245k sequence length ladder (chunk sizes 128..2048). Measured min PCC >= 0.9994 (threshold >= 0.984). Output recorded in bench/runs/chunked-sdpa-position-scaling-20260917T192147Z.jsonl and commit d40a764.
  • E2E A/B Fastbench / Stage 3: PASS.
    • Control (QWEN36_SDPA_STREAMING=0): bench/runs/sdpa-streaming-e2e-control-20260919T123737Z.jsonl
    • Treatment (QWEN36_SDPA_STREAMING=1): bench/runs/sdpa-streaming-e2e-treatment-20260919T131333Z.jsonl
    • Verdict recorded in bench/runs/sdpa-streaming-e2e-verdict-20260919T1430Z.jsonl (commit 45a39de).
  • Topology: 2x Tenstorrent Blackhole p150a, TP=2, 93,739-token prompt context, warm cache.
  • Measured Metrics:
    • Makespan: conc-1 -1.56%, conc-4 -2.07%, conc-8 -2.15% (301.2s -> 294.7s).
    • TTFT (p50): conc-1 -2.11%, conc-4 -2.38%, conc-8 -2.42% (max TTFT -6.6s).
    • Decode TG ms/tok: unchanged / flat within noise (conc-1 -0.19%, conc-4 -0.29%, conc-8 +0.58%).
    • Prefill PP tok/s: conc-8 2,489.6 -> 2,544.3 tok/s (+2.20%).
  • Ship Recommendation: Shipped as default sdpaStreaming = true in Nix module and production recipe via PR #44.
Completed and integrated in PR #44. ## Evidence Summary: - **PCC / Correctness Gate**: PASS. `bench/probes/sdpa_streaming_pcc_probe.py` evaluated across 245k sequence length ladder (chunk sizes 128..2048). Measured min PCC >= 0.9994 (threshold >= 0.984). Output recorded in `bench/runs/chunked-sdpa-position-scaling-20260917T192147Z.jsonl` and commit `d40a764`. - **E2E A/B Fastbench / Stage 3**: PASS. - Control (`QWEN36_SDPA_STREAMING=0`): `bench/runs/sdpa-streaming-e2e-control-20260919T123737Z.jsonl` - Treatment (`QWEN36_SDPA_STREAMING=1`): `bench/runs/sdpa-streaming-e2e-treatment-20260919T131333Z.jsonl` - Verdict recorded in `bench/runs/sdpa-streaming-e2e-verdict-20260919T1430Z.jsonl` (commit `45a39de`). - **Topology**: 2x Tenstorrent Blackhole p150a, TP=2, 93,739-token prompt context, warm cache. - **Measured Metrics**: - Makespan: conc-1 -1.56%, conc-4 -2.07%, conc-8 -2.15% (301.2s -> 294.7s). - TTFT (p50): conc-1 -2.11%, conc-4 -2.38%, conc-8 -2.42% (max TTFT -6.6s). - Decode TG ms/tok: unchanged / flat within noise (conc-1 -0.19%, conc-4 -0.29%, conc-8 +0.58%). - Prefill PP tok/s: conc-8 2,489.6 -> 2,544.3 tok/s (+2.20%). - **Ship Recommendation**: Shipped as default `sdpaStreaming = true` in Nix module and production recipe via PR #44.
Grok closed this issue 2026-09-19 16:16:08 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
ttx
mtp-thinkingcap
tt-quant
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#35
No description provided.