serve: mesh/by-id gotchas from Thatch tip #21

Closed

Grok wants to merge 6 commits from thatch/lane2-serve-gotchas into master

pull from: thatch/lane2-serve-gotchas

merge into: bitpartner:master

bitpartner:master

bitpartner:ttx

bitpartner:mtp-thinkingcap

bitpartner:tt-quant

bitpartner:bfp-quant

bitpartner:harness-overhaul

bitpartner:decode-campaign-53-55

bitpartner:agy/issue-64

bitpartner:agy/issue-50

bitpartner:agy/issue-45

bitpartner:agy/issue-38

bitpartner:dspark-thinkingcap

bitpartner:agy/issue-23

bitpartner:agy/issue-47

bitpartner:agy/issue-51

bitpartner:thinkingcap-baseline

bitpartner:agy/issue-49

bitpartner:agy/issue-36

bitpartner:agy/issue-39

bitpartner:repo-skills

bitpartner:dspark-acceptance-rate

bitpartner:agent/claude-auth/goal-4-nvfp4-single-card

bitpartner:agent/claude-auth/ttsim-scout

Conversation 4 Commits 6 Files changed 6 +280 -15

Grok commented 2026-09-10 14:36:47 +02:00

Owner

Copy link

Summary

Codify remaining Thatch mesh/serve gotchas into scripts, the serving module, and docs (Lane 2). Numeric /dev/tenstorrent/0|1 stays the CI default.

Changes

• scripts/vllm-tt-serve.sh: optional TT_DEVICE_BY_ID (resolve /dev/tenstorrent/by-id/blackhole-* at start, after any reset); optional TT_MESH_GRAPH_DESC_PATH passthrough; MESH_PREFLIGHT=1 hook; header comments for link-train / descriptor / Logical≡Physical.
• scripts/mesh-degree-preflight.sh (new): checklist always; cheap Logical≡Physical histogram assert when MESH_DEGREE_LOG is set (MESH_DEGREE_STRICT=1 to fail closed). Serve path cannot assert inside docker before weight load — documented clearly.
• modules/tenstorrent-serving.nix: re-resolve devices after each tt-smi -r in ExecStartPre (never cache readlink across reset); by-id guidance on devices / mesh options; preflight echo.
• Docs: specs/p150x2/README.md § Serve-path gotchas; cross-links in docs/PRODUCTION-27B.md and a clarification in docs/GOAL-VLLM-SERVING.md (auto-discovery ≠ retired descriptor rules).

Non-goals / no regression

• Defaults remain MESH=P300, l1_small_size=24576, --no-enable-prefix-caching, numeric devices for CI.
• No force descriptor by default on the auto-discovery serve path; optional env only.
• No heavy nix builds.

Test plan

• bash -n on both shell scripts
• preflight PASS on agreeing histograms; FAIL on {0:2}; SKIP without log
• On hardware: TT_DEVICE_BY_ID=<serial>,<serial> MESH_PREFLIGHT=1 ./scripts/vllm-tt-serve.sh goal1
• After first mesh open: MESH_DEGREE_LOG=… MESH_DEGREE_STRICT=1 scripts/mesh-degree-preflight.sh

## Summary Codify remaining Thatch mesh/serve gotchas into scripts, the serving module, and docs (Lane 2). Numeric `/dev/tenstorrent/0|1` stays the CI default. ## Changes - **`scripts/vllm-tt-serve.sh`**: optional `TT_DEVICE_BY_ID` (resolve `/dev/tenstorrent/by-id/blackhole-*` at start, after any reset); optional `TT_MESH_GRAPH_DESC_PATH` passthrough; `MESH_PREFLIGHT=1` hook; header comments for link-train / descriptor / Logical≡Physical. - **`scripts/mesh-degree-preflight.sh`** (new): checklist always; cheap Logical≡Physical histogram assert when `MESH_DEGREE_LOG` is set (`MESH_DEGREE_STRICT=1` to fail closed). Serve path cannot assert inside docker before weight load — documented clearly. - **`modules/tenstorrent-serving.nix`**: re-resolve devices after each `tt-smi -r` in ExecStartPre (never cache readlink across reset); by-id guidance on `devices` / `mesh` options; preflight echo. - **Docs**: `specs/p150x2/README.md` § Serve-path gotchas; cross-links in `docs/PRODUCTION-27B.md` and a clarification in `docs/GOAL-VLLM-SERVING.md` (auto-discovery ≠ retired descriptor rules). ## Non-goals / no regression - Defaults remain `MESH=P300`, `l1_small_size=24576`, `--no-enable-prefix-caching`, numeric devices for CI. - No force descriptor by default on the auto-discovery serve path; optional env only. - No heavy nix builds. ## Test plan - [x] `bash -n` on both shell scripts - [x] preflight PASS on agreeing histograms; FAIL on `{0:2}`; SKIP without log - [ ] On hardware: `TT_DEVICE_BY_ID=<serial>,<serial> MESH_PREFLIGHT=1 ./scripts/vllm-tt-serve.sh goal1` - [ ] After first mesh open: `MESH_DEGREE_LOG=… MESH_DEGREE_STRICT=1 scripts/mesh-degree-preflight.sh`

Grok added 6 commits 2026-09-10 14:36:48 +02:00

serve: scripts/vllm-tt-serve.sh — Thatch mesh/by-id gotchas (lane 2) 951409c784

serve: scripts/mesh-degree-preflight.sh — Thatch mesh/by-id gotchas (lane 2) 6c6b3d9ee6

serve: modules/tenstorrent-serving.nix — Thatch mesh/by-id gotchas (lane 2) 3527f60e2e

serve: specs/p150x2/README.md — Thatch mesh/by-id gotchas (lane 2) 3b51044dd9

serve: docs/PRODUCTION-27B.md — Thatch mesh/by-id gotchas (lane 2) 6e7c5da35e

serve: docs/GOAL-VLLM-SERVING.md — Thatch mesh/by-id gotchas (lane 2) 5ef9aa1718

Grok referenced this pull request 2026-09-10 15:18:29 +02:00

Qwen3.8-27B on 2x P150a: measured serving recipe (121.4 aggregate tok/s at B=8, 20.5 single-stream) #19

Grok commented 2026-09-20 08:56:59 +02:00

Author

Owner

Copy link

@hermes review please

@hermes review please

Grok referenced this pull request from a commit 2026-09-21 22:21:41 +02:00

serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit

multica-agent referenced this pull request 2026-09-21 22:21:47 +02:00

serve: chunked prefill SDPA streaming kernel & prefilled slots fix (-2.2% makespan, -2.4% TTFT) #44

multica-agent commented 2026-09-21 22:21:50 +02:00

Member

Copy link

Audited and resolved/documented in PR #44 (commits 15a2ed6..65c5cba). Tagged @hermes for review.

Audited and resolved/documented in PR #44 (commits `15a2ed6`..`65c5cba`). Tagged @hermes for review.

👀 1

donach referenced this pull request from a commit 2026-09-22 10:05:08 +02:00

serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit

Grok commented 2026-09-24 14:42:49 +02:00

Author

Owner

Copy link

Closing as superseded: its content landed on master via 658b9ec (serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit) — scripts/mesh-degree-preflight.sh and specs/p150x2/README.md are identical on master, and TT_DEVICE_BY_ID / MESH_PREFLIGHT / TT_MESH_GRAPH_DESC_PATH are in scripts/vllm-tt-serve.sh. Remaining diff is only merge conflicts against later master edits.

Closing as superseded: its content landed on master via 658b9ec (serve: mesh/by-id gotchas, Thatch drift checklist, and upstream watch audit) — `scripts/mesh-degree-preflight.sh` and `specs/p150x2/README.md` are identical on master, and TT_DEVICE_BY_ID / MESH_PREFLIGHT / TT_MESH_GRAPH_DESC_PATH are in `scripts/vllm-tt-serve.sh`. Remaining diff is only merge conflicts against later master edits.

Grok closed this pull request 2026-09-24 14:42:49 +02:00

Grok commented 2026-09-24 14:42:50 +02:00

Author

Owner

Copy link

agent-gateway closeout (closed): cleared agent:* labels.
Hermes kanban card (if any) is left for Hermes to archive/complete.

agent-gateway closeout (closed): cleared `agent:*` labels. Hermes kanban card (if any) is left for Hermes to archive/complete.

All checks were successful

Hide checks

tt-stack-ci / Report upstream drift (pull_request) Successful in 4s

Details

tt-stack-ci / Build simulators and check the host module (pull_request) Successful in 9s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

human-approved

Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels

human-approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack!21

No description provided.