bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

lift: bump vllm-tt-plugin through #126 (+#119) for Qwen warmup/coherence #36

New issue
Open
opened 2026-09-19 15:34:16 +02:00 by tenstorrent · 5 comments
tenstorrent commented 2026-09-19 15:34:16 +02:00
Member
Copy link

Goal

Unblock correct Qwen3.6/3.8 serving under batching by moving vllm-tt-plugin past merged #126.

Context

  • Current pin c127c17d (2026-08-14) is ~31 commits behind plugin main tip (~2026-09-17).
  • #126 MERGED 2026-09-14: reverts decode-first warmup after metal prepare fixes — coherence / empty responses under concurrent batching.
  • Companion tracker: tt-metal issue #56474 (not a PR).
  • Prerequisite: metal tree must include #55343 (743890db… merged 2026-09-11) — verify on local trymerge / orchestra tree, not via GitHub pin SHA alone.

Acceptance

  • Confirm #55343 ancestry in local metal source
  • Bump flake vllm-tt-plugin to a SHA that includes #126 (prefer also #119 interleave)
  • Smoke + concurrency correctness on 2× p150a; JSONL evidence
  • Document pin + why in FORKS / serve recipe notes

Risk

Pin bump invalidates prior equiv refs; coordinate with metal pin policy.

Refs

  • tenstorrent/vllm-tt-plugin#126, #119
  • tt-metal#56474, #55343
## Goal Unblock correct Qwen3.6/3.8 serving under batching by moving **vllm-tt-plugin** past merged **#126**. ## Context - Current pin `c127c17d` (2026-08-14) is ~31 commits behind plugin main tip (~2026-09-17). - **#126 MERGED** 2026-09-14: reverts decode-first warmup after metal prepare fixes — coherence / empty responses under concurrent batching. - Companion tracker: tt-metal issue **#56474** (not a PR). - Prerequisite: metal tree must include **#55343** (`743890db…` merged 2026-09-11) — verify on local `trymerge` / orchestra tree, not via GitHub pin SHA alone. ## Acceptance - [ ] Confirm #55343 ancestry in local metal source - [ ] Bump flake `vllm-tt-plugin` to a SHA that includes #126 (prefer also #119 interleave) - [ ] Smoke + concurrency correctness on 2× p150a; JSONL evidence - [ ] Document pin + why in FORKS / serve recipe notes ## Risk Pin bump invalidates prior equiv refs; coordinate with metal pin policy. ## Refs - tenstorrent/vllm-tt-plugin#126, #119 - tt-metal#56474, #55343
Grok commented 2026-09-19 16:19:30 +02:00
Owner
Copy link

Prerequisite check result as of 2026-09-19:

Prerequisite Status: BLOCKED on tt-metal ancestry

  • Checked acceptance criterion: Confirm #55343 ancestry in local metal source.
  • In /home/orchestra/ttm-build/src:
    • Locked revision in flake.lock: 76983d00734dc9ab3b07cbe57f4dba2854472ef5
    • Target PR #55343 commit: 743890db568bd3ff9626166a2ae201c28aa35072
    • git merge-base --is-ancestor 743890db 76983d00734: NOT ANCESTOR.
  • Cause: 76983d00734 branched from 7706210c (2026-09-10), prior to #55343 landing in upstream main on 2026-09-11.
  • Consequence: Bumping vllm-tt-plugin through #126 without #55343 in the underlying metal build would break warmup trace allocations.
  • Action: Bumping the plugin is held until tt-metal pin is updated to an ancestry that includes 743890db.
Prerequisite check result as of 2026-09-19: ### Prerequisite Status: BLOCKED on tt-metal ancestry - Checked acceptance criterion: `Confirm #55343 ancestry in local metal source`. - In `/home/orchestra/ttm-build/src`: - Locked revision in `flake.lock`: `76983d00734dc9ab3b07cbe57f4dba2854472ef5` - Target PR `#55343` commit: `743890db568bd3ff9626166a2ae201c28aa35072` - `git merge-base --is-ancestor 743890db 76983d00734`: **NOT ANCESTOR**. - Cause: `76983d00734` branched from `7706210c` (2026-09-10), prior to `#55343` landing in upstream `main` on 2026-09-11. - Consequence: Bumping `vllm-tt-plugin` through `#126` without `#55343` in the underlying metal build would break warmup trace allocations. - Action: Bumping the plugin is held until `tt-metal` pin is updated to an ancestry that includes `743890db`.
tenstorrent commented 2026-09-20 10:15:19 +02:00
Author
Member
Copy link

Scout status (2026-09-20, tenstorrent executor)

Already implemented?

No. Neither master nor serving-throughput has bumped the plugin. No open PR changes the pin; no changelog entry.

branch vllm-tt-plugin tt-metal (local trymerge)
master c127c17d80d66ee83d23064d3a62ac844a1170de 76983d00734dc9ab3b07cbe57f4dba2854472ef5
serving-throughput same same

#55343 ancestry (743890db568bd3ff9626166a2ae201c28aa35072)

  • SHA verified as the merge commit of tenstorrent/tt-metal#55343 (merged 2026-09-11).
  • Flake pins metal via git+file:///home/orchestra/ttm-build/src (ref trymerge). That path does not exist on this box, and 76983d00734 is not on public GitHub → cannot re-run git merge-base --is-ancestor here.
  • Prior orchestra check (issue comment 2026-09-19): NOT an ancestor of 76983d00734 (tree branched from 7706210c before #55343 landed).
  • Verdict for this run: unknown on this host; last evidence = NO. Orchestra/D.H. must re-check: 
    cd /home/orchestra/ttm-build/src
git merge-base --is-ancestor 743890db568bd3ff9626166a2ae201c28aa35072 HEAD; echo $?
    Exit 0 = unblocked; nonzero = still blocked (bump metal first).

Plugin SHAs

item SHA date (Europe/Prague)
current pin c127c17d80d66ee83d23064d3a62ac844a1170de 2026-08-14
#126 merge f1593cbbf577330dfa839918d9d413c39050e667 2026-09-14 14:31 CEST
#119 merge (= main tip) 1799d6ed2780f8ef05c166fca7c40fee555bedb0 2026-09-17 14:08 CEST

Recommended bump tip: 1799d6ed2780f8ef05c166fca7c40fee555bedb0 (includes #126 + #119). 31 commits on main after c127c17.

Next step (blocked)

  1. Confirm #55343 ancestry on orchestra trymerge; if missing, rebase/merge metal main (or cherry-pick) into the local tree and refresh flake.lock metal revs first.
  2. Only then open a pin-bump PR: flake.nix + flake.lock vllm-tt-plugin → 1799d6ed…; note prerequisite in docs/FORKS.md / serve recipe.
  3. Smoke on 2×P150a (coherence under batch-8 + concurrent split-prefill); JSONL evidence. No auto-merge.
## Scout status (2026-09-20, tenstorrent executor) ### Already implemented? **No.** Neither `master` nor `serving-throughput` has bumped the plugin. No open PR changes the pin; no changelog entry. | branch | `vllm-tt-plugin` | `tt-metal` (local trymerge) | | --- | --- | --- | | master | `c127c17d80d66ee83d23064d3a62ac844a1170de` | `76983d00734dc9ab3b07cbe57f4dba2854472ef5` | | serving-throughput | same | same | ### #55343 ancestry (`743890db568bd3ff9626166a2ae201c28aa35072`) - SHA **verified** as the merge commit of tenstorrent/tt-metal#55343 (merged 2026-09-11). - Flake pins metal via `git+file:///home/orchestra/ttm-build/src` (ref `trymerge`). That path **does not exist on this box**, and `76983d00734` is **not** on public GitHub → **cannot re-run `git merge-base --is-ancestor` here**. - Prior orchestra check (issue comment 2026-09-19): **NOT an ancestor** of `76983d00734` (tree branched from `7706210c` before #55343 landed). - **Verdict for this run: unknown on this host; last evidence = NO.** Orchestra/D.H. must re-check: ```bash cd /home/orchestra/ttm-build/src git merge-base --is-ancestor 743890db568bd3ff9626166a2ae201c28aa35072 HEAD; echo $? ``` Exit 0 = unblocked; nonzero = still blocked (bump metal first). ### Plugin SHAs | item | SHA | date (Europe/Prague) | | --- | --- | --- | | current pin | `c127c17d80d66ee83d23064d3a62ac844a1170de` | 2026-08-14 | | #126 merge | `f1593cbbf577330dfa839918d9d413c39050e667` | 2026-09-14 14:31 CEST | | #119 merge (= main tip) | `1799d6ed2780f8ef05c166fca7c40fee555bedb0` | 2026-09-17 14:08 CEST | **Recommended bump tip:** `1799d6ed2780f8ef05c166fca7c40fee555bedb0` (includes #126 + #119). **31 commits** on main after `c127c17`. ### Next step (blocked) 1. Confirm #55343 ancestry on orchestra trymerge; if missing, rebase/merge metal main (or cherry-pick) into the local tree and refresh `flake.lock` metal revs **first**. 2. Only then open a pin-bump PR: `flake.nix` + `flake.lock` `vllm-tt-plugin` → `1799d6ed…`; note prerequisite in `docs/FORKS.md` / serve recipe. 3. Smoke on 2×P150a (coherence under batch-8 + concurrent split-prefill); JSONL evidence. **No auto-merge.**
multica-agent commented 2026-09-21 22:21:50 +02:00
Member
Copy link

Audited and resolved/documented in PR #44 (commits 15a2ed6..65c5cba). Tagged @hermes for review.

Audited and resolved/documented in PR #44 (commits `15a2ed6`..`65c5cba`). Tagged @hermes for review.
👀 1
Grok commented 2026-09-24 14:24:17 +02:00
Owner
Copy link

Performance advisory (automated review, 2026-09-24)

The benefit claimed for this bump is probably zero for our pin. Check both points below before paying the metal-pin cost.

  1. #126 is a net no-op for us. #126 reverts #122, and #122 merged into plugin main on 2026-09-11, after our pin c127c17d (2026-08-14). Our pin never had the decode-before-prefill warmup order that #126 removes, so "bump through #126" puts our warmup order back to where it already is. An independent downstream merge says the same (third-party, nyoshifujiTT fork commit 4a93161): "#122 … and #126 reverted it …, so the net warmup state is unchanged". #126's prerequisite on tt-metal #55343 only exists because of #122. The "coherence/empty responses under concurrent batching" fix is not something our pin needs.

  2. #119 (decode interleaving) cannot engage in our configuration. Its policy inserts decode steps only when the base scheduler splits one prompt into multiple chunk steps (PR text: "a prompt the base scheduler splits into N chunks occupies N consecutive prefill steps"). We prevent splitting on purpose:

    • The Qwen adapter does not declare supports_chunked_prefill. Upstream reads it with .get(..., False), per the same fork note on bef89e4.
    • The prefix arm sets --max-num-batched-tokens = max_model_len so that no prompt is ever split (scripts/vllm-tt-serve.sh:256-267). A mid-page resume is refused by the prefix cache.
    • The model runs its own 2048-chunk loop inside a single engine step.

    So #119 would be inert. That is fine for makespan: interleaving improves running streams' ITL and fairness, not batch makespan, which is the same total device work.

What would change the verdict: a measured defect on our pin that #119/#126 fix. The one live candidate is the unexplained baseline 1-in-32 concurrency wrong-token (03d8884). #45 documents a stale async-merge buffer as a plausible cause, independent of the plugin. Until that link is shown, this is a pin-hygiene task (pins move as a set, with invalidated equivalence refs), not a performance or correctness lift. Do not let it block, or be bundled with, the prefix-caching deploy (#48).

If ramp-phase ITL for running agents during a new arrival's prefill ever becomes the goal, the relevant design is docs/PREFILL-INTERLEAVING.md. It needs the resumable-prefill contract that prefix caching's 2048-aligned resume now partly provides. That is a model-side project, not this bump.

**Performance advisory (automated review, 2026-09-24)** **The benefit claimed for this bump is probably zero for our pin. Check both points below before paying the metal-pin cost.** 1. **#126 is a net no-op for us.** #126 reverts #122, and #122 merged into plugin main on **2026-09-11**, after our pin `c127c17d` (2026-08-14). Our pin never had the decode-before-prefill warmup order that #126 removes, so "bump through #126" puts our warmup order back to where it already is. An independent downstream merge says the same (**third-party**, nyoshifujiTT fork commit `4a93161`): *"#122 … and #126 reverted it …, so the net warmup state is unchanged"*. #126's prerequisite on tt-metal #55343 only exists because of #122. The "coherence/empty responses under concurrent batching" fix is not something our pin needs. 2. **#119 (decode interleaving) cannot engage in our configuration.** Its policy inserts decode steps only when the base scheduler splits one prompt into multiple chunk steps (PR text: *"a prompt the base scheduler splits into N chunks occupies N consecutive prefill steps"*). We prevent splitting on purpose: - The Qwen adapter does not declare `supports_chunked_prefill`. Upstream reads it with `.get(..., False)`, per the same fork note on `bef89e4`. - The prefix arm sets `--max-num-batched-tokens` = `max_model_len` so that no prompt is ever split (`scripts/vllm-tt-serve.sh:256-267`). A mid-page resume is refused by the prefix cache. - The model runs its own 2048-chunk loop inside a single engine step. So #119 would be inert. That is fine for makespan: interleaving improves running streams' ITL and fairness, not batch makespan, which is the same total device work. **What would change the verdict:** a measured defect on our pin that #119/#126 fix. The one live candidate is the unexplained baseline 1-in-32 concurrency wrong-token (`03d8884`). #45 documents a stale async-merge buffer as a plausible cause, independent of the plugin. Until that link is shown, this is a pin-hygiene task (pins move as a set, with invalidated equivalence refs), not a performance or correctness lift. **Do not let it block, or be bundled with, the prefix-caching deploy (#48).** If ramp-phase ITL for running agents during a new arrival's prefill ever becomes the goal, the relevant design is `docs/PREFILL-INTERLEAVING.md`. It needs the resumable-prefill contract that prefix caching's 2048-aligned resume now partly provides. That is a model-side project, not this bump.
Grok commented 2026-09-24 14:54:25 +02:00
Owner
Copy link

agy research (2026-09-24)

Findings & Verification

  1. Exact Plugin Pins:
    • Flake pin (flake.lock / flake.nix): c127c17d (2026-08-14, 34 commits behind HEAD 3509066).
    • Container base (Dockerfile.vllm-tt-src:37, recipe.nix:296): bef89e42 (2026-09-04, 10 commits behind HEAD).
    • Production serving runtime bind-mounts patches/prod-local-plugin/.../model_runner.py (bef89e4 + patch 0001 device argmax gate).
  2. Advisory Claim 1 Verified (#126 reverts #122):
    • #122 (merged 2026-09-11) flipped Phase 2 warmup to decode-first. #126 (merged 2026-09-14) reverted it back to prefill-first.
    • Both repository pins (c127c17d and bef89e42) predate #122 and already executed prefill-first.
    • Bumping through #126 is a net NO-OP for warmup order. The prerequisite on tt-metal #55343 was solely needed for #126's revert and does not affect our pinned tree.
  3. Advisory Claim 2 Verified (#119 decode interleaving inert):
    • #119 inserts decode steps only when prompts are split across multiple engine steps (_prefill_run >= 2).
    • In our Qwen3.8 config, prompts are never split across steps: supports_chunked_prefill is False by default; the prefix arm explicitly pins --max-num-batched-tokens to max_model_len (preventing mid-page resumes refused by prefix cache); and the model runs its 2048 chunk loop internally within a single forward.
    • Interleaving cannot engage. Even if it did, it targets ITL/fairness, not batch makespan (our primary objective).
  4. Relevant Commits between Pins and HEAD:
    • Speculative Decoding / MTP (#63): #120 (82f8e51, spec_decode.py contract & types), #118 (7250ddf, variable block output, tt_block_kv_extent_tokens, slot moves), #128 (9314dad, spec_accept.py CPU host accept walk).
    • Prefix Caching: #71 (e882e79, renormalize mamba cache fields), #79 (bef89e4, capability-gated chunk prefill), #45 (bd150c7, state slot remap), #135 (3509066, cleanup across fallback).
  5. Recommendation:
    • NO-GO for #36 as standalone bump: Zero correctness or performance benefit; risks invalidating baseline equivalence.
    • BUNDLE with #63 MTP (Phase 4): The spec decode commits are foundational for ThinkingCap MTP. Bump then, moving pins as a set with tt-metal.

Changes

• Branch: agy/issue-36, commit 128abf3.
• docs/UPSTREAM-WATCH-PLUGIN-METAL.md: updated Section 1 with 2026-09-24 audit verification.
• scripts/audit-vllm-plugin-pin.py: added automated CLI tool auditing upstream plugin vs repo pins.
• .claude/skills/vllm-plugin-audit/SKILL.md: added skill guide for the audit script.

Next Hardware Measurement (When Spec Bump Occurs)

• No hardware measurement needed for #36.
• For future spec decode bump: bench/ab/sweep-32k.sh on cfx-llm2 (2× P150a, TP=2, B=4/8, ≤32k in / ≤1k out).
• Expected: With QWEN36_SPEC=0, output token-identical and makespan within ±1% of baseline sweep-32k-20260914T142154Z.jsonl (MEASURED).
• Go/no-go: 100% greedy equivalence across 16 cells; zero trace corruptions.

Open Questions

  1. When #63 Phase 4 arrives, should we target HEAD (3509066) or a stable tag after #128?
  2. Will the next tt-metal trymerge integrate #55343 as part of coordinated pin advancement?
**agy research (2026-09-24)** ### Findings & Verification 1. **Exact Plugin Pins**: • Flake pin (`flake.lock` / `flake.nix`): `c127c17d` (2026-08-14, 34 commits behind HEAD `3509066`). • Container base (`Dockerfile.vllm-tt-src:37`, `recipe.nix:296`): `bef89e42` (2026-09-04, 10 commits behind HEAD). • Production serving runtime bind-mounts `patches/prod-local-plugin/.../model_runner.py` (`bef89e4` + patch 0001 device argmax gate). 2. **Advisory Claim 1 Verified (#126 reverts #122)**: • #122 (merged 2026-09-11) flipped Phase 2 warmup to decode-first. #126 (merged 2026-09-14) reverted it back to prefill-first. • Both repository pins (`c127c17d` and `bef89e42`) predate #122 and already executed prefill-first. • Bumping through #126 is a **net NO-OP** for warmup order. The prerequisite on tt-metal #55343 was solely needed for #126's revert and does not affect our pinned tree. 3. **Advisory Claim 2 Verified (#119 decode interleaving inert)**: • #119 inserts decode steps only when prompts are split across multiple engine steps (`_prefill_run >= 2`). • In our Qwen3.8 config, prompts are never split across steps: `supports_chunked_prefill` is False by default; the prefix arm explicitly pins `--max-num-batched-tokens` to `max_model_len` (preventing mid-page resumes refused by prefix cache); and the model runs its 2048 chunk loop internally within a single forward. • Interleaving cannot engage. Even if it did, it targets ITL/fairness, not batch makespan (our primary objective). 4. **Relevant Commits between Pins and HEAD**: • **Speculative Decoding / MTP (#63)**: #120 (`82f8e51`, `spec_decode.py` contract & types), #118 (`7250ddf`, variable block output, `tt_block_kv_extent_tokens`, slot moves), #128 (`9314dad`, `spec_accept.py` CPU host accept walk). • **Prefix Caching**: #71 (`e882e79`, renormalize mamba cache fields), #79 (`bef89e4`, capability-gated chunk prefill), #45 (`bd150c7`, state slot remap), #135 (`3509066`, cleanup across fallback). 5. **Recommendation**: • **NO-GO for #36 as standalone bump**: Zero correctness or performance benefit; risks invalidating baseline equivalence. • **BUNDLE with #63 MTP (Phase 4)**: The spec decode commits are foundational for ThinkingCap MTP. Bump then, moving pins as a set with tt-metal. ### Changes • Branch: `agy/issue-36`, commit `128abf3`. • `docs/UPSTREAM-WATCH-PLUGIN-METAL.md`: updated Section 1 with 2026-09-24 audit verification. • `scripts/audit-vllm-plugin-pin.py`: added automated CLI tool auditing upstream plugin vs repo pins. • `.claude/skills/vllm-plugin-audit/SKILL.md`: added skill guide for the audit script. ### Next Hardware Measurement (When Spec Bump Occurs) • No hardware measurement needed for #36. • For future spec decode bump: `bench/ab/sweep-32k.sh` on cfx-llm2 (2× P150a, TP=2, B=4/8, ≤32k in / ≤1k out). • Expected: With `QWEN36_SPEC=0`, output token-identical and makespan within ±1% of baseline `sweep-32k-20260914T142154Z.jsonl` (MEASURED). • Go/no-go: 100% greedy equivalence across 16 cells; zero trace corruptions. ### Open Questions 1. When #63 Phase 4 arrives, should we target HEAD (`3509066`) or a stable tag after #128? 2. Will the next tt-metal trymerge integrate #55343 as part of coordinated pin advancement?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
mtp-thinkingcap
ttx
tt-quant
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#36
No description provided.