docs: split system from stack so iteration stops needing a deploy #7

Merged

multica-agent merged 1 commit from agent/claude-auth/on-box-iteration into master

2026-09-04 14:14:28 +02:00

multica-agent commented

2026-09-04 14:14:09 +02:00

Member

Operator direction: stack work happens on the box, in containers, and a NixOS deploy should not be needed for most changes.

That is also what upstream does — tt-installer puts driver, firmware and system tools on the host and installs TT-Metalium as a Podman container by default. And it is what modules/tenstorrent-host.nix already claims: its job "ends at 'the cards are present, usable, and correctly permissioned'". The boundary was right; the container workflow above it was the missing half.

Only four things still need a deploy (first three also need a reboot): cardCount/hugepagesPerCard (kernel command line — a switch cannot apply them), the udev rule / device group, the model-store mount, and a tt-kmd bump. Everything else — tt-metal rev, patch stack, vLLM flags, model code, kernels, checkpoints — is a container rebuild.

The case is stronger than convenience: a failed unit makes switch-to-configuration exit 4, deploy-rs rolls the generation back, and the CI deploy job reddens for the whole cluster. That happened three times in one afternoon (an inventory unit asserting before the host had booted; a mount pointing at an unformatted disk) and none of them were stack changes.

Also records the incremental-rebuild trick that makes kernel iteration viable — build the patch layer FROM the built base image, ~a minute instead of hours — which matters because items 2–4 of SATURATION-PLAN.md are all kernel changes.

Verified on cfx-llm2 today: docker present and active, podman not installed. ReBAR/Above-4G are working — Region 4: … [size=32G], the full GDDR6 aperture. And one BIOS requirement we had missed entirely: PCIe AER Reporting must be OS First because tt-smi depends on it, and it must be re-set after any BIOS update. Noted that the nixos host boots with pci=noaer and that pattern must not be copied onto a TT host.

Docs only.

Operator direction: stack work happens **on the box**, in containers, and a NixOS deploy should not be needed for most changes. That is also what upstream does — `tt-installer` puts driver, firmware and system tools on the host and installs **TT-Metalium as a Podman container by default**. And it is what `modules/tenstorrent-host.nix` already claims: its job "ends at 'the cards are present, usable, and correctly permissioned'". The boundary was right; the container workflow above it was the missing half. **Only four things still need a deploy** (first three also need a reboot): `cardCount`/`hugepagesPerCard` (kernel command line — a `switch` cannot apply them), the udev rule / device group, the model-store mount, and a tt-kmd bump. Everything else — tt-metal rev, patch stack, vLLM flags, model code, kernels, checkpoints — is a container rebuild. The case is stronger than convenience: a failed unit makes `switch-to-configuration` exit 4, deploy-rs rolls the generation back, and the CI deploy job reddens for the whole cluster. That happened **three times in one afternoon** (an inventory unit asserting before the host had booted; a mount pointing at an unformatted disk) and none of them were stack changes. Also records the incremental-rebuild trick that makes kernel iteration viable — build the patch layer FROM the built base image, ~a minute instead of hours — which matters because items 2–4 of `SATURATION-PLAN.md` are all kernel changes. **Verified on cfx-llm2 today:** `docker` present and active, `podman` **not** installed. ReBAR/Above-4G are working — `Region 4: … [size=32G]`, the full GDDR6 aperture. And one BIOS requirement we had missed entirely: **PCIe AER Reporting must be `OS First`** because tt-smi depends on it, and it must be re-set after any BIOS update. Noted that the **nixos** host boots with `pci=noaer` and that pattern must not be copied onto a TT host. Docs only.

multica-agent added 1 commit

2026-09-04 14:14:09 +02:00

docs: split system from stack so iteration stops needing a deploy

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 6s

Details

c4f0ffc95e

Operator direction: stack work happens on the box, in containers, and a NixOS
deploy should not be needed for most changes.

That matches what upstream does — tt-installer puts driver, firmware and
system tools on the host and installs TT-Metalium as a Podman container by
default — and it matches what modules/tenstorrent-host.nix already claims,
that its job ends at "the cards are present, usable, and correctly
permissioned". The boundary was right; the container workflow above it was
missing.

Only four things now require a deploy, and the first three also require a
reboot: cardCount / hugepagesPerCard (kernel command line, so a switch
cannot apply them), the udev rule and device group, the model-store mount,
and a tt-kmd bump. Everything else — tt-metal rev, patch stack, vLLM flags,
model code, kernels, checkpoints — is a container rebuild.

The argument for doing this is stronger than convenience. A failed unit makes
switch-to-configuration exit 4, deploy-rs rolls the generation back, and the
CI deploy job reddens for the whole cluster. That happened three times in one
afternoon — an inventory unit asserting before the host had booted, and a
mount pointing at an unformatted disk — and none of them were stack changes.
Moving the stack off the deploy path removes the coupling.

Records the container invocation with the load-bearing parts (device resolved
AFTER a reset, the 1G hugetlbfs bind that IS the device-to-host aperture,
SYS_NICE, TT_MESH_GRAPH_DESC_PATH pointing at p300 not p150_x2, and
127.0.0.1-only port publishing), and the incremental-rebuild trick that makes
kernel work ~a minute instead of hours: build the patch layer FROM the built
base image.

Verified on cfx-llm2 today: docker present and active, podman NOT installed.
And two BIOS facts — ReBAR/Above-4G are working (Region 4 reports size=32G,
the full GDDR6 aperture), plus a requirement we had missed entirely: PCIe AER
Reporting must be "OS First" because tt-smi depends on it, and it must be
re-set after any BIOS update. Noted that the nixos host boots with pci=noaer
and that pattern must not be copied onto a TT host.

multica-agent merged commit ee5a21f08d into master

2026-09-04 14:14:28 +02:00

multica-agent referenced this pull request from a commit

2026-09-04 14:14:29 +02:00

Merge pull request 'docs: split system from stack so iteration stops needing a deploy' (#7) from agent/claude-auth/on-box-iteration into master

multica-agent referenced this pull request

2026-09-11 15:29:38 +02:00

Qwen3.8-27B on 2x P150a: measured serving recipe (121.4 aggregate tok/s at B=8, 20.5 single-stream) #19