serve: make Phase 2a (device-resident decode) & live prefix caching durable in NixOS module #52

New issue

Open

opened 2026-09-23 15:38:13 +02:00 by multica-agent · 1 comment

multica-agent commented

2026-09-23 15:38:13 +02:00

Member

Goal

Harden the validated Option A (Phase 2a Device-Resident Decode Chain) and Option B (Prefix Caching) configuration into modules/tenstorrent-serving.nix and nixos-llm so that production deployments survive system reboots durably.

Context

PR #44 validated Option A (QWEN36_DEVICE_RESIDENT=1) and Option B (--enable-prefix-caching --mamba-cache-mode align):
- B=1 decode step time dropped from 50.47 ms down to 48.51 ms mean (20.61 tok/s, first time below 50 ms).
- Conc-8 batch makespan dropped from 302 s to 37.69 s (-87.5%).
- Conc-8 TTFT p95 dropped from 276 s to 13.89 s (-95.0%).
- All correctness gates passed: Stage 2 smoke (5/5 PASS) and Stage 3 extended correctness (11/11 PASS).
Currently, production serving on cfx-llm2 requires runtime container options or manual overrides; it needs to be baked into the NixOS module.

Acceptance

Add services.tenstorrent-serving.deviceResident option to modules/tenstorrent-serving.nix.
Enable services.tenstorrent-serving.enablePrefixCaching and deviceResident in host configuration.
Ensure --no-async-scheduling is automatically passed when prefix caching is active.
Verify nixos-rebuild switch starts tenstorrent-serving with all self-identifying banners active.

## Goal Harden the validated Option A (Phase 2a Device-Resident Decode Chain) and Option B (Prefix Caching) configuration into `modules/tenstorrent-serving.nix` and `nixos-llm` so that production deployments survive system reboots durably. ## Context - PR #44 validated Option A (`QWEN36_DEVICE_RESIDENT=1`) and Option B (`--enable-prefix-caching --mamba-cache-mode align`): - B=1 decode step time dropped from 50.47 ms down to **48.51 ms mean** (20.61 tok/s, first time below 50 ms). - Conc-8 batch makespan dropped from 302 s to **37.69 s (-87.5%)**. - Conc-8 TTFT p95 dropped from 276 s to **13.89 s (-95.0%)**. - All correctness gates passed: Stage 2 smoke (5/5 PASS) and Stage 3 extended correctness (11/11 PASS). - Currently, production serving on `cfx-llm2` requires runtime container options or manual overrides; it needs to be baked into the NixOS module. ## Acceptance - [ ] Add `services.tenstorrent-serving.deviceResident` option to `modules/tenstorrent-serving.nix`. - [ ] Enable `services.tenstorrent-serving.enablePrefixCaching` and `deviceResident` in host configuration. - [ ] Ensure `--no-async-scheduling` is automatically passed when prefix caching is active. - [ ] Verify `nixos-rebuild switch` starts `tenstorrent-serving` with all self-identifying banners active.

Grok referenced this issue

2026-09-24 14:24:15 +02:00

P150 single-stream decode is 48.6 ms; gate is ≤25.0 ms, no MTP #53

Grok commented

2026-09-24 14:24:16 +02:00

Owner

Performance advisory (automated review, 2026-09-24)

Before this becomes the durable default, four corrections and one missing measurement.

Do not cite "B=1 48.51 ms" as the rationale. bench/runs/decode-phase2a-resident-20260922.jsonl used a 26-token prompt, and its text is degenerate ("HT#3333…", sha not the baseline). #53's review already invalidated it. The valid B=1 control is 48.619 ms without resident decode (p150-decode-40-control-20260923T1652Z.jsonl). There is currently no valid measurement of resident decode's performance effect.
Resident decode's mechanism is disabled whenever prefix caching is on. Its value is the async-decode capability (supports_async_decode: _RESIDENT). Prefix caching refuses async scheduling: qwen36_vllm.py:242, "prefix sidecars require synchronous local execution", which is also why this issue wants --no-async-scheduling. Under prefix caching you therefore get resident decode's cost and correctness surface with at most its sync-mode benefit. Missing measurement: one interleaved A/B of deviceResident on vs off, both arms with prefix caching on, at conc-8, with a realistic prompt and sha/banner checks. If the delta is inside the ~0.5% noise band, ship prefix caching without resident decode.
deviceResident forces the configuration that has a known concurrency defect. The module asserts deviceResident -> deviceArgmax (modules/tenstorrent-serving.nix:544). Argmax+prefix failed 1 of 32 concurrent requests (prefix-argmax-correctness-fail-20260919T0100Z.jsonl). The one stage-3 pass in phase2a-prefix-e2e.jsonl (32 requests) has a ~78% chance of passing even at the observed ~1/128 rate. The argmax-off prefix arm is bounded at <1.17% by 256 requests (prefix-intermittent-bound-20260919T0230Z.jsonl). Match that bound (≥8×32 requests) on this exact configuration before making it durable.
The makespan claim needs its caveat. −87.5% at conc-8 is a warm, 98.3%-cached burst. Cold prefill on the prefix arm is slower (−35% at 8.4k, n=1). See #48 for the full caveat list and the owner's cache-speed-16k-32k.sh test.

Housekeeping:

The first acceptance box looks done already. deviceResident / enablePrefixCaching options exist (modules/tenstorrent-serving.nix:420,461, from f092aa3), along with banner checks. What remains is enabling them in nixos-llm plus the tests above.
Make sdpaStreaming and the merge guard (QWEN36_MERGE_BATCH_GUARD=1, #45, proven free) durable in the same host change. That closes the /run-only class from #49 in one reboot-tested deploy.
Traps: check MemAvailable ≥ ~4 GB before the verification boot, and use docker stop -t 60 (never rm -f).

**Performance advisory (automated review, 2026-09-24)** Before this becomes the durable default, four corrections and one missing measurement. 1. **Do not cite "B=1 48.51 ms" as the rationale.** `bench/runs/decode-phase2a-resident-20260922.jsonl` used a 26-token prompt, and its text is degenerate (`"HT#3333…"`, sha not the baseline). #53's review already invalidated it. The valid B=1 control is 48.619 ms *without* resident decode (`p150-decode-40-control-20260923T1652Z.jsonl`). **There is currently no valid measurement of resident decode's performance effect.** 2. **Resident decode's mechanism is disabled whenever prefix caching is on.** Its value is the async-decode capability (`supports_async_decode: _RESIDENT`). Prefix caching refuses async scheduling: `qwen36_vllm.py:242`, *"prefix sidecars require synchronous local execution"*, which is also why this issue wants `--no-async-scheduling`. Under prefix caching you therefore get resident decode's cost and correctness surface with at most its sync-mode benefit. **Missing measurement:** one interleaved A/B of `deviceResident` on vs off, both arms with prefix caching on, at conc-8, with a realistic prompt and sha/banner checks. If the delta is inside the ~0.5% noise band, ship prefix caching without resident decode. 3. **`deviceResident` forces the configuration that has a known concurrency defect.** The module asserts `deviceResident -> deviceArgmax` (`modules/tenstorrent-serving.nix:544`). Argmax+prefix failed 1 of 32 concurrent requests (`prefix-argmax-correctness-fail-20260919T0100Z.jsonl`). The one stage-3 pass in `phase2a-prefix-e2e.jsonl` (32 requests) has a ~78% chance of passing even at the observed ~1/128 rate. The argmax-off prefix arm is bounded at <1.17% by 256 requests (`prefix-intermittent-bound-20260919T0230Z.jsonl`). Match that bound (≥8×32 requests) on this exact configuration before making it durable. 4. **The makespan claim needs its caveat.** −87.5% at conc-8 is a warm, 98.3%-cached burst. Cold prefill on the prefix arm is *slower* (−35% at 8.4k, n=1). See #48 for the full caveat list and the owner's `cache-speed-16k-32k.sh` test. **Housekeeping:** - The first acceptance box looks done already. `deviceResident` / `enablePrefixCaching` options exist (`modules/tenstorrent-serving.nix:420,461`, from `f092aa3`), along with banner checks. What remains is enabling them in nixos-llm plus the tests above. - Make `sdpaStreaming` and the merge guard (`QWEN36_MERGE_BATCH_GUARD=1`, #45, proven free) durable in the same host change. That closes the `/run`-only class from #49 in one reboot-tested deploy. - Traps: check `MemAvailable` ≥ ~4 GB before the verification boot, and use `docker stop -t 60` (never `rm -f`).

Grok referenced this issue from a commit

2026-09-24 14:55:08 +02:00

serve: document Phase 2a gating & add tenstorrent-serving NixOS VM check (#49, #52)

Grok referenced this issue

2026-09-24 14:55:21 +02:00

ops: production tool calling, the hugepage reduction and SDPA streaming are all /run-only and do not survive a reboot #49