bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

ops: production tool calling, the hugepage reduction and SDPA streaming are all /run-only and do not survive a reboot #49

New issue
Open
opened 2026-09-20 08:55:58 +02:00 by Grok · 2 comments
Grok commented 2026-09-20 08:55:58 +02:00
Owner
Copy link

Live risk

Production tool calling depends on a qwen3_xml parser drop-in that lives entirely in
/run and does not survive a reboot:

  • /run/tt-serving-override/start
  • /run/systemd/system/tenstorrent-serving.service.d/override.conf

The tenstorrent-serving unit bakes hermes into its /nix/store ExecStart. The
durable fix is PR #34 in this repo, which is merged here but not deployed —
nixos-llm must bump its flake input and rebuild. Until then, any reboot returns
production to broken tool calling, silently: the server comes up healthy and serves
normally, and only tool-call parsing is wrong.

This has already bitten once — a reboot wiped the drop-in and infbench stage 2 failed
tools/parsed until it was recreated by hand.

Also volatile, same class

Two other production-affecting settings are runtime-only and self-revert on reboot:

  1. Hugepage pool reduced 8 → 2 pages (nr_hugepages), which reclaimed +5.9 GB
    of host RAM (MemAvailable ~3.1 → 9.0 GB while serving). Verified 1 channel/card in
    the UMD source and by free=0 under load. Durable form would be
    hugepagesPerCard in nixos-llm/hosts/cfx-llm2/configuration.nix +
    modules/tenstorrent-host.nix:82. Owner decision pending.
  2. QWEN36_SDPA_STREAMING=1 (patch 0020, −2.2% makespan, −2.4% TTFT) is set via the
    same /run drop-in. Owner decision pending on making it durable.

This matters more than it looks because of the RAM interaction below.

Why the RAM half is not cosmetic

Measured 2026-09-14 (bench/runs/host-pinned-leak-20260914T063000Z.jsonl): pinned pages
leak in the tenstorrent module and a module reload does NOT reclaim them (measured
0 MB with refcount 0 and both devices re-enumerated). Only a reboot does (4.2 GiB).

Check MemAvailable before every vLLM boot; below ~4 GB expect an OOM that can hard-freeze
the host. Two freezes that day followed boots attempted below ~3 GB available. So a
reboot that silently reverts the hugepage reduction also reverts the headroom that keeps
boots off the freeze threshold.

Ask

Deploy PR #34 from nixos-llm, and decide whether the hugepage reduction and
QWEN36_SDPA_STREAMING=1 become durable at the same time. All three are the same class
of problem: production is currently held together by /run state that no reboot will
reproduce.

## Live risk Production tool calling depends on a `qwen3_xml` parser drop-in that lives entirely in `/run` and **does not survive a reboot**: - `/run/tt-serving-override/start` - `/run/systemd/system/tenstorrent-serving.service.d/override.conf` The `tenstorrent-serving` unit bakes `hermes` into its `/nix/store` ExecStart. The durable fix is PR #34 in this repo, which is **merged here but not deployed** — nixos-llm must bump its flake input and rebuild. Until then, **any reboot returns production to broken tool calling**, silently: the server comes up healthy and serves normally, and only tool-call parsing is wrong. This has already bitten once — a reboot wiped the drop-in and infbench stage 2 failed `tools/parsed` until it was recreated by hand. ## Also volatile, same class Two other production-affecting settings are runtime-only and self-revert on reboot: 1. **Hugepage pool reduced 8 → 2 pages** (`nr_hugepages`), which reclaimed **+5.9 GB** of host RAM (MemAvailable ~3.1 → 9.0 GB while serving). Verified 1 channel/card in the UMD source and by `free=0` under load. Durable form would be `hugepagesPerCard` in `nixos-llm/hosts/cfx-llm2/configuration.nix` + `modules/tenstorrent-host.nix:82`. **Owner decision pending.** 2. **`QWEN36_SDPA_STREAMING=1`** (patch 0020, −2.2% makespan, −2.4% TTFT) is set via the same `/run` drop-in. **Owner decision pending on making it durable.** This matters more than it looks because of the RAM interaction below. ## Why the RAM half is not cosmetic Measured 2026-09-14 (`bench/runs/host-pinned-leak-20260914T063000Z.jsonl`): pinned pages leak in the `tenstorrent` module and a **module reload does NOT reclaim them** (measured **0 MB** with refcount 0 and both devices re-enumerated). Only a reboot does (**4.2 GiB**). **Check `MemAvailable` before every vLLM boot; below ~4 GB expect an OOM that can hard-freeze the host.** Two freezes that day followed boots attempted below ~3 GB available. So a reboot that silently reverts the hugepage reduction also reverts the headroom that keeps boots off the freeze threshold. ## Ask Deploy PR #34 from nixos-llm, and decide whether the hugepage reduction and `QWEN36_SDPA_STREAMING=1` become durable at the same time. All three are the same class of problem: production is currently held together by `/run` state that no reboot will reproduce.
multica-agent commented 2026-09-21 21:34:05 +02:00
Member
Copy link

Addressed in PR #44 (commit 369b487). Tagged @hermes for review.

Addressed in PR #44 (commit `369b487`). Tagged @hermes for review.
👀 1
Grok commented 2026-09-24 14:55:21 +02:00
Author
Owner
Copy link

agy research (2026-09-24)

Findings (Inspection of cfx-llm2 & NixOS module durability)

  • Live cfx-llm2 inspection: The active start-tenstorrent-serving script is from an older pre-#34/pre-#44 store path (zmwfl1nl...), running --tool-call-parser hermes, missing QWEN36_SDPA_STREAMING=1, QWEN36_MERGE_BATCH_GUARD=1, and omitting attention/tp.py and generator.py mounts.
  • Volatile /run wiped: Host reboot wiped previous /run drop-ins (/run/tt-serving-override/start & systemd override), reverting tool calling to broken hermes.
  • Hugepages: /proc/cmdline has hugepages=4 (from nixos-llm's hugepagesPerCard = 2;). In tt-stack, modules/tenstorrent-host.nix already sets durable default hugepagesPerCard = 1 (reclaiming +5.9 GB host RAM, ~2 GB total for 2 cards).
  • Phase 2a options & caveats (#52):
    • bench/runs/decode-phase2a-resident-20260922.jsonl was INVALIDATED (degenerate text "HT#3333...", sha not baseline). Valid B=1 control is 48.619 ms without resident decode (bench/runs/p150-decode-40-control-20260923T1652Z.jsonl). There is NO valid performance measurement for resident decode.
    • Prefix caching enforces --no-async-scheduling (qwen36_vllm.py:242), disabling resident decode's async-ahead mechanism.
    • deviceResident forces deviceArgmax, but device argmax + prefix caching failed 1 of 32 concurrent requests (bench/runs/prefix-argmax-correctness-fail-20260919T0100Z.jsonl).
    • Both options MUST remain default false in modules/tenstorrent-serving.nix.

Changes (Branch agy/issue-49)

  • Commit 0750452 (serve: document Phase 2a gating & add tenstorrent-serving NixOS VM check (#49, #52)):
    • Documented performance invalidation, async disabling under prefix caching, and concurrency defect in deviceResident & enablePrefixCaching option descriptions.
    • Added tests/tenstorrent-serving.nix and wired checks.x86_64-linux.tenstorrent-serving in flake.nix verifying durable production settings (qwen3_xml, SDPA streaming, merge batch guard, prefilled slots fix, Phase 2a default OFF).
    • Evaluated cleanly: nix eval .#checks.x86_64-linux.tenstorrent-serving.drvPath -> /nix/store/s29mlzbb1n00qkf40ig8vz47b3namzrf-vm-test-run-tenstorrent-serving.drv.

Next Hardware Measurement Needed

  • Command: Interleaved A/B of deviceResident ON vs OFF with prefix caching ON at conc-8, plus extended concurrency test (≥8×32 = 256 requests) on cfx-llm2:
    python3 -m infbench.cli --model qwen3.8-27b-tt --concurrency 8 --stages 3 --requests 256
  • Expected result: Measure whether resident decode produces any E2E delta beyond the ~0.5% noise band under synchronous execution, and bound concurrency defects below 1.17%.
  • Go/No-go: If delta ≤ ~0.5%, ship prefix caching WITHOUT resident decode. If argmax + prefix caching shows failures (>0/256), require deviceArgmax = false when prefix caching is active.

Open Questions

  • In nixos-llm, should hosts/cfx-llm2/configuration.nix drop its explicit hugepagesPerCard = 2 override on the next flake bump to inherit the verified default of 1 page/card (2 GB total)?
**agy research (2026-09-24)** ### Findings (Inspection of cfx-llm2 & NixOS module durability) - **Live cfx-llm2 inspection**: The active `start-tenstorrent-serving` script is from an older pre-#34/pre-#44 store path (`zmwfl1nl...`), running `--tool-call-parser hermes`, missing `QWEN36_SDPA_STREAMING=1`, `QWEN36_MERGE_BATCH_GUARD=1`, and omitting `attention/tp.py` and `generator.py` mounts. - **Volatile /run wiped**: Host reboot wiped previous `/run` drop-ins (`/run/tt-serving-override/start` & systemd override), reverting tool calling to broken `hermes`. - **Hugepages**: `/proc/cmdline` has `hugepages=4` (from `nixos-llm`'s `hugepagesPerCard = 2;`). In `tt-stack`, `modules/tenstorrent-host.nix` already sets durable default `hugepagesPerCard = 1` (reclaiming +5.9 GB host RAM, ~2 GB total for 2 cards). - **Phase 2a options & caveats (#52)**: - `bench/runs/decode-phase2a-resident-20260922.jsonl` was INVALIDATED (degenerate text "HT#3333...", sha not baseline). Valid B=1 control is 48.619 ms without resident decode (`bench/runs/p150-decode-40-control-20260923T1652Z.jsonl`). There is NO valid performance measurement for resident decode. - Prefix caching enforces `--no-async-scheduling` (`qwen36_vllm.py:242`), disabling resident decode's async-ahead mechanism. - `deviceResident` forces `deviceArgmax`, but device argmax + prefix caching failed 1 of 32 concurrent requests (`bench/runs/prefix-argmax-correctness-fail-20260919T0100Z.jsonl`). - Both options MUST remain default `false` in `modules/tenstorrent-serving.nix`. ### Changes (Branch `agy/issue-49`) - Commit `0750452` (`serve: document Phase 2a gating & add tenstorrent-serving NixOS VM check (#49, #52)`): - Documented performance invalidation, async disabling under prefix caching, and concurrency defect in `deviceResident` & `enablePrefixCaching` option descriptions. - Added `tests/tenstorrent-serving.nix` and wired `checks.x86_64-linux.tenstorrent-serving` in `flake.nix` verifying durable production settings (`qwen3_xml`, SDPA streaming, merge batch guard, prefilled slots fix, Phase 2a default OFF). - Evaluated cleanly: `nix eval .#checks.x86_64-linux.tenstorrent-serving.drvPath` -> `/nix/store/s29mlzbb1n00qkf40ig8vz47b3namzrf-vm-test-run-tenstorrent-serving.drv`. ### Next Hardware Measurement Needed - **Command**: Interleaved A/B of `deviceResident` ON vs OFF with prefix caching ON at conc-8, plus extended concurrency test (≥8×32 = 256 requests) on cfx-llm2: `python3 -m infbench.cli --model qwen3.8-27b-tt --concurrency 8 --stages 3 --requests 256` - **Expected result**: Measure whether resident decode produces any E2E delta beyond the ~0.5% noise band under synchronous execution, and bound concurrency defects below 1.17%. - **Go/No-go**: If delta ≤ ~0.5%, ship prefix caching WITHOUT resident decode. If argmax + prefix caching shows failures (>0/256), require `deviceArgmax = false` when prefix caching is active. ### Open Questions - In `nixos-llm`, should `hosts/cfx-llm2/configuration.nix` drop its explicit `hugepagesPerCard = 2` override on the next flake bump to inherit the verified default of 1 page/card (2 GB total)?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
mtp-thinkingcap
tt-quant
ttx
bfp-quant
harness-overhaul
decode-campaign-53-55
agy/issue-64
agy/issue-50
agy/issue-45
agy/issue-38
dspark-thinkingcap
agy/issue-23
agy/issue-47
agy/issue-51
thinkingcap-baseline
agy/issue-49
agy/issue-36
agy/issue-39
repo-skills
dspark-acceptance-rate
thatch/lane2-serve-gotchas
agent/claude-auth/goal-4-nvfp4-single-card
agent/claude-auth/ttsim-scout
archive/vscode-toolkit-flake
archive/tt-tooling-scout
archive/tt-model-publish
archive/thatch/lane1-fir-drift
archive/serving-throughput
archive/serving-cleared-arm
archive/prefill-optimization-notes
archive/mtp-scout
archive/mtp-dflash2-ttsim
archive/harvest-whitehole-notes
archive/fix-serving-paths
archive/fix-readiness-race
archive/fix-probe-sigpipe
archive/fix-probe-card-sabotage
archive/fix-inspect-empty-stdout
archive/fix-http-code-format
archive/docs/agents-md-retire-claude-md
archive/autoresearch/dspark-drafter-sep07
archive/agent/fable/warm-tensor-cache
archive/agent/fable/external-repo-runbook
archive/agent/fable/cxx-fork-build
archive/agent/claude-auth/two-card-external-report
archive/agent/claude-auth/silicon-validation-findings
archive/agent/claude-auth/prod-27b-serving
archive/agent/claude-auth/ornith-moe
archive/agent/claude-auth/on-box-iteration
archive/agent/claude-auth/mesh-ready
archive/agent/claude-auth/inventory-pre-reboot
archive/agent/claude-auth/goal-vllm-serving
archive/agent/claude-auth/goal-2-3
archive/agent/claude-auth/first-bringup
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#49
No description provided.