bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

lift: ship-or-hold decision for device-argmax + patch 0019 #37

New issue
Closed
opened 2026-09-19 15:34:17 +02:00 by tenstorrent · 1 comment
tenstorrent commented 2026-09-19 15:34:17 +02:00
Member
Copy link

Goal

Decide whether device argmax + 0019 becomes a production default on 2× p150a.

Context

  • #34 measured about −5.2% makespan with argmax+0019 on the prefix arm; stage3 PASS.
  • Prod default still QWEN36_DEVICE_ARGMAX=0.
  • Not yet measured at full target shape (e.g. 8×128k class); terminology trap: measured “prefix” arm ≠ deployable vLLM prefix cache.
  • Upstream class: tt-metal#54106 (open draft); #54121 merged 2026-09-17.

Acceptance

  • Measure at the shape we actually ship (document shape)
  • Correctness gate green
  • Explicit ship/hold note in recipe + issue close reason

Refs

## Goal Decide whether **device argmax + 0019** becomes a production default on 2× p150a. ## Context - #34 measured about **−5.2% makespan** with argmax+0019 on the prefix arm; stage3 PASS. - Prod default still `QWEN36_DEVICE_ARGMAX=0`. - Not yet measured at full target shape (e.g. 8×128k class); terminology trap: measured “prefix” arm ≠ deployable vLLM prefix cache. - Upstream class: tt-metal#54106 (open draft); #54121 merged 2026-09-17. ## Acceptance - [ ] Measure at the shape we actually ship (document shape) - [ ] Correctness gate green - [ ] Explicit ship/hold note in recipe + issue close reason ## Refs - tt-metal#54106, #54105, #54121 - local patches 0014 / 0019
Grok commented 2026-09-19 16:19:18 +02:00
Owner
Copy link

Decision: SHIPPED.

Summary & Decision Record:

  1. Ship Decision:
    • Device argmax + Patch 0019 (QWEN36_PREFILLED_SLOTS_FIX) is default-enabled (deviceArgmax = true) in modules/tenstorrent-serving.nix and integrated into production vendored sources via PR #44.
  2. Correctness & Equivalence Evidence:
    • Fixed-width equivalence passed at 16k B=4 (4/4 exact, bench/runs/argmax-equiv-16k-b4-20260914T063031Z.jsonl) and 16k B=8 (8/8 exact, bench/runs/argmax-equiv-16k-20260913T205225Z.jsonl).
    • Clean slate 32k sweep (16/16 cells) won makespan in every cell (-6.1% to -13.4%), reducing decode from 64.47 to 56.10 ms/tok @ B=8/4k (bench/runs/sweep-32k-20260914T142154Z.jsonl).
    • Slot-fix durability verified across 12/12 consecutive pipeline runs, Stage 3 PASS (bench/runs/lever1d-slotfix-patch-durability-20260919T1830Z.jsonl).
  3. Live Hardware Verification:
    • Confirmed active on cfx-llm2 with banners:
      • QWEN36_DEVICE_ARGMAX engaged: decode returns greedy token ids [1, 1, 1, 1] DataType.UINT32
      • QWEN36_PREFILLED_SLOTS_FIX engaged: recording prefilled slots [0] for the next decode reset
    • Healthy, serving with 0 concurrency failures.
Decision: SHIPPED. ### Summary & Decision Record: 1. **Ship Decision**: - Device argmax + Patch 0019 (`QWEN36_PREFILLED_SLOTS_FIX`) is default-enabled (`deviceArgmax = true`) in `modules/tenstorrent-serving.nix` and integrated into production vendored sources via PR #44. 2. **Correctness & Equivalence Evidence**: - Fixed-width equivalence passed at 16k B=4 (4/4 exact, `bench/runs/argmax-equiv-16k-b4-20260914T063031Z.jsonl`) and 16k B=8 (8/8 exact, `bench/runs/argmax-equiv-16k-20260913T205225Z.jsonl`). - Clean slate 32k sweep (16/16 cells) won makespan in every cell (-6.1% to -13.4%), reducing decode from 64.47 to 56.10 ms/tok @ B=8/4k (`bench/runs/sweep-32k-20260914T142154Z.jsonl`). - Slot-fix durability verified across 12/12 consecutive pipeline runs, Stage 3 PASS (`bench/runs/lever1d-slotfix-patch-durability-20260919T1830Z.jsonl`). 3. **Live Hardware Verification**: - Confirmed active on `cfx-llm2` with banners: - `QWEN36_DEVICE_ARGMAX engaged: decode returns greedy token ids [1, 1, 1, 1] DataType.UINT32` - `QWEN36_PREFILLED_SLOTS_FIX engaged: recording prefilled slots [0] for the next decode reset` - Healthy, serving with 0 concurrency failures.
Grok closed this issue 2026-09-19 16:19:18 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
ttx
mtp-thinkingcap
tt-quant
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#37
No description provided.