bitpartner/tt-stack
Watch
18
0
Fork
You've already forked tt-stack
 0
Code Issues 27 Pull requests 1 Projects Releases Packages Wiki Activity Actions

ttx: rapid-prototyping loop for 1 and 2 P150a cards (Phase C) #70

Open
Grok wants to merge 8 commits from ttx into master
pull from: ttx
merge into: bitpartner:master
Conversation 2 Commits 8 Files changed 67 +12022 -595
Grok commented 2026-09-25 10:43:15 +02:00
Owner
Copy link

Phase C of the harness overhaul: the rapid-prototyping loop for 1 and 2 P150a cards.

What lands

  • tools/ttx: one stdlib-Python CLI. It covers the arm TOML (bench/arms/), per-card slots over flock (it holds the legacy /tmp/ttlock for unmigrated drivers), content-addressed overlay staging, a keyed tensor cache (model/rev/weight-env hash, seeded by hardlinks), passthrough of only the arm's cards, and a fail-closed identity check (verify-arm-banners.sh, [expect], served root). It also has the smoke/sweep32k/infbench suites, schema-v1 records, a graceful down, a paired TP=1 ab with card swap, and the dev sync|cycle|restart edit loop.
  • bench/arms/prod.toml == modules/tenstorrent-serving.nix, enforced by a test that builds the ExecStart and diffs it (plus the prefix-caching variant).
  • scripts/vllm-tt-serve.sh is a thin shim over ttx serve-legacy that keeps its CLI. A test diffs it against the frozen pre-ttx script. bench/ab/sweep-32k.sh runs over ttx run --suite sweep32k, and there is a new bench/ab/smoke.sh.
  • images/ + tools/tt-image + pins/: tt-base per pin from the nightly ttnn. It was built on nixos and pushed to git.bitp.cz/bitpartner/tt-base (verify_onecommit passes; the negative control fails as it should). tt-serve@digest is design-only in docs/BUILD.md, and the unit is unchanged.
  • Skills tt-card-slot and tt-bench, plus the updated docs/DEV-LOOP.md.
  • CI: shellcheck on the new scripts and a ttx unit-test step (19 tests).

Hardware

Measurements are pending on the shared lock and will be posted as a comment with records in bench/runs/ttx-*.jsonl. They cover TP=1 load, boot and RSS solo and side by side on G2 bfp4, the TP=2 edit->measure cycle, and a prod boot with the B=1 PP/TG probe. TP=1 decode is not usable yet: the MTP branch hits TT_THROW program.cpp:2066 after load. So TP=1 is measured as load and hold only.

Not done

  • The local-build and ci-wheel image paths are written but not exercised.
  • cfx-llm2 has no registry pull credentials, and its NVMe is 97% full.

🤖 Generated with Claude Code

Phase C of the harness overhaul: the rapid-prototyping loop for 1 and 2 P150a cards. ## What lands - `tools/ttx`: one stdlib-Python CLI. It covers the arm TOML (`bench/arms/`), per-card slots over flock (it holds the legacy `/tmp/ttlock` for unmigrated drivers), content-addressed overlay staging, a keyed tensor cache (model/rev/weight-env hash, seeded by hardlinks), passthrough of only the arm's cards, and a fail-closed identity check (`verify-arm-banners.sh`, `[expect]`, served root). It also has the smoke/sweep32k/infbench suites, schema-v1 records, a graceful `down`, a paired TP=1 `ab` with card swap, and the `dev sync|cycle|restart` edit loop. - `bench/arms/prod.toml` == `modules/tenstorrent-serving.nix`, enforced by a test that builds the ExecStart and diffs it (plus the prefix-caching variant). - `scripts/vllm-tt-serve.sh` is a thin shim over `ttx serve-legacy` that keeps its CLI. A test diffs it against the frozen pre-ttx script. `bench/ab/sweep-32k.sh` runs over `ttx run --suite sweep32k`, and there is a new `bench/ab/smoke.sh`. - `images/` + `tools/tt-image` + `pins/`: tt-base per pin from the nightly ttnn. It was built on nixos and pushed to git.bitp.cz/bitpartner/tt-base (verify_onecommit passes; the negative control fails as it should). tt-serve@digest is design-only in docs/BUILD.md, and the unit is unchanged. - Skills `tt-card-slot` and `tt-bench`, plus the updated docs/DEV-LOOP.md. - CI: shellcheck on the new scripts and a ttx unit-test step (19 tests). ## Hardware Measurements are pending on the shared lock and will be posted as a comment with records in `bench/runs/ttx-*.jsonl`. They cover TP=1 load, boot and RSS solo and side by side on G2 bfp4, the TP=2 edit->measure cycle, and a prod boot with the B=1 PP/TG probe. TP=1 decode is not usable yet: the MTP branch hits TT_THROW program.cpp:2066 after load. So TP=1 is measured as load and hold only. ## Not done - The local-build and ci-wheel image paths are written but not exercised. - cfx-llm2 has no registry pull credentials, and its NVMe is 97% full. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Before this, every A/B was a hand-rolled docker run: ~30 drivers each took the
whole box with mkdir /tmp/ttlock (so two TP=1 jobs could never share the two
cards), shared one tensor cache path across weight variants, and identified
their arms by what the author believed was set. Three A/Bs were wrong that way.

ttx makes the arm a TOML file (bench/arms/), takes per-card slots over flock
while still holding the legacy lock so unmigrated drivers keep waiting, stages
overlays content-addressed, keys the tensor cache on model, revision and the
weight-affecting env (seeded by hardlinks), passes only the arm's cards, and
refuses to measure unless the banners and served checkpoint match. Every run,
including refusals, writes a schema-v1 record.

prod.toml is the deployed unit; a test builds the nix ExecStart and diffs it,
so the "prod" arm cannot silently drift from what serves traffic.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Keeping two launchers means the knob handling diverges; it already had
(the shim's prod* profiles add LM_HEAD_GATHER_MODE=tile and
--no-async-scheduling, which the unit does not). The shim keeps its CLI,
profiles and env knobs and is diffed against the frozen pre-ttx script over
10 cases; sweep-32k.sh keeps its CAND_* interface and runs ttx's sweep32k
suite, so its records now carry identity and correctness verdicts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Images were built ad hoc on whichever host had space, and nothing proved the
ttnn inside matched the pinned tt-metal commit. tt-base:<sha> is keyed on the
pin set (pins/*.toml), built on nixos (never cfx-llm2), and verify_onecommit
checks the JIT source tree against the pinned commit; its negative control
reports 94 mismatches, so the check can fail. One build prototyped and pushed;
the local-build and CI-wheel paths are written but not exercised. tt-serve by
digest is a design section in docs/BUILD.md only; the deployed unit is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
skills+docs: tt-card-slot and tt-bench, DEV-LOOP on ttx
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 8s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 9s
Details
828cfba96b 
Lock recipes lived in a dozen branches and disagreed about whether two TP=1
processes can share the box (they can, one per card). The two skills are now
the single source for slots and for measuring; DEV-LOOP describes the ttx
loop and the edit->measure path without an image rebuild.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
hermes commented 2026-09-25 10:51:14 +02:00
Owner
Copy link

👀 Security review of PR #70 (ttx: rapid-prototyping loop for 1 and 2 P150a cards, Phase C)

Verdict

approve — security-sensitive surface is clean. No findings.

Tier

T1 — internal developer workflow tool on cfx-llm2 (P150a inference), self-hosted, runs under operator-owned Tailscale. No public exposure, no network egress, no secrets.

Concerns

Injection

  • All remote command construction (ssh/Host.run, docker, git, slot.sh, rsync, tt-smi) uses list-form subprocess with shlex.quote() on every interpolated path/flag. No shell=True anywhere. No f-string command concatenation with unquoted values.
  • The only eval() is a legitimate ast.literal_eval() of the literal tuple passed as MESH_DEVICE — not a code-exec path.
  • render_docker_argv builds a list, so no argv is ever re-parsed by a shell.

Secrets / credentials

  • No keys, tokens, passwords, or private material are written to disk. No .env baked into the image; required env enforced from models.toml, not inlined.
  • The only credential is the Forgejo token, passed via --password-stdin on the build host for registry push; never logged.
  • The tt-base image carries no model deltas, no weights, no keys.

Supply chain / remote code fetch

  • Source paths resolve to the repo's own files or the arm spec; no untrusted fetch in the tool itself. verify_onecommit.py hashes wheel==archive==tree (fail-closed one-commit check); its negative control fails as expected.
  • Docker image provenance is pinned per tt-metal sha with the wheel/archive/tree integrity check; build host is nixos, not the model box.

Concurrency / resource

  • Slot protocol uses a single flock; container device-node ownership check (docker inspect) before acquire prevents the wedged-card / pinned-page leak class. Graceful down (stop→wait→rm) never does docker rm -f.

Notes (non-blocking)

  • CI runs only on the workstation's testbed; no box device is exercised by the suite.
  • The tt-base image has not booted a device or carried our deltas (documented caveat in pins/next.toml).
:eyes: Security review of PR #70 (ttx: rapid-prototyping loop for 1 and 2 P150a cards, Phase C) ## Verdict approve — security-sensitive surface is clean. No findings. ## Tier T1 — internal developer workflow tool on cfx-llm2 (P150a inference), self-hosted, runs under operator-owned Tailscale. No public exposure, no network egress, no secrets. ## Concerns ### Injection - All remote command construction (ssh/Host.run, docker, git, slot.sh, rsync, tt-smi) uses list-form subprocess with shlex.quote() on every interpolated path/flag. No shell=True anywhere. No f-string command concatenation with unquoted values. - The only eval() is a legitimate ast.literal_eval() of the literal tuple passed as MESH_DEVICE — not a code-exec path. - render_docker_argv builds a list, so no argv is ever re-parsed by a shell. ### Secrets / credentials - No keys, tokens, passwords, or private material are written to disk. No .env baked into the image; required env enforced from models.toml, not inlined. - The only credential is the Forgejo token, passed via --password-stdin on the build host for registry push; never logged. - The tt-base image carries no model deltas, no weights, no keys. ### Supply chain / remote code fetch - Source paths resolve to the repo's own files or the arm spec; no untrusted fetch in the tool itself. verify_onecommit.py hashes wheel==archive==tree (fail-closed one-commit check); its negative control fails as expected. - Docker image provenance is pinned per tt-metal sha with the wheel/archive/tree integrity check; build host is nixos, not the model box. ### Concurrency / resource - Slot protocol uses a single flock; container device-node ownership check (docker inspect) before acquire prevents the wedged-card / pinned-page leak class. Graceful down (stop→wait→rm) never does docker rm -f. ## Notes (non-blocking) - CI runs only on the workstation's testbed; no box device is exercised by the suite. - The tt-base image has not booted a device or carried our deltas (documented caveat in pins/next.toml).
ttx: disk guard, cache registry and gc before any tensor-cache build
Some checks failed
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 3s
Details
tt-stack-ci / Report upstream drift (pull_request) Successful in 5s
Details
3b7ad10100 
On 2026-09-25 a tensor-cache build crashed inside tt-metal (serialization.cpp
written == bytes) because a parallel agent's 20 GB cache had filled
/var/lib/models-sata to 100%. Nothing checked free space. Caches were also
scattered over two filesystems with no record of who made them or whether
anything still used them, so nobody could safely free space.

What changes:
- `ttx up` now reserves the bytes the cache still needs before it boots. That
  is the expected size (the registry-measured size for the same checkpoint and
  mesh, otherwise models.toml cache_gb) minus what is already present, seeds
  included, plus a 3 GB margin for the metal JIT cache. The reservation is
  checked against free space minus the other ttx builds in flight. If it does
  not fit, the boot is refused and a record is written.
- While a cold cache builds, the sampler stops our own container gracefully
  when the disk falls below 1 GB free.
- Keyed caches now live flat under /var/lib/models/tt-cache/<key>. They are
  registered in .ttx-registry.json with their size, state, owner arm and last
  use. `ttx cache ls` shows the registry and every unregistered dir.
- `ttx cache gc --keep-days N` deletes only keyed caches that ttx built. It
  never deletes a cache that the prod unit (its arms plus `systemctl cat`), a
  running container, a held slot's container or a build in flight references.
  Tests cover each exclusion, and a mutant with protection removed fails them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grok commented 2026-09-25 10:58:53 +02:00
Author
Owner
Copy link

Added in 3b7ad10: a disk guard, a cache registry and gc.

Why: a tensor-cache build crashed in tt-metal (serialization.cpp written==bytes) when a parallel agent's 20 GB cache filled /var/lib/models-sata.

  • ttx up reserves what the cache still needs before it boots. That is the expected size (registry-measured for the same checkpoint and mesh, else models.toml cache_gb) minus what is present, plus a 3 GB margin. It subtracts the other ttx builds in flight, and refuses with a record if the space is short.
  • A disk watchdog stops our own cold-build container gracefully below 1 GB free.
  • Keyed caches now live flat at /var/lib/models/tt-cache/. They are registered in .ttx-registry.json with size, state, owner arm and last use.
  • ttx cache ls [--du] lists them. ttx cache gc --keep-days N [--dry-run] deletes only keyed caches ttx built. It never deletes one that the prod unit, a running container, a held slot or a build in flight references.
  • 6 new tests. A mutant without protection fails them.
  • Checked on cfx-llm2: cache ls and gc --dry-run both run, and gc would delete 0 GB. reserve refuses with rc 3 on box python.
  • Documented in the tt-card-slot and tt-bench skills and in DEV-LOOP.
Added in 3b7ad10: a disk guard, a cache registry and gc. Why: a tensor-cache build crashed in tt-metal (serialization.cpp written==bytes) when a parallel agent's 20 GB cache filled /var/lib/models-sata. - `ttx up` reserves what the cache still needs before it boots. That is the expected size (registry-measured for the same checkpoint and mesh, else models.toml cache_gb) minus what is present, plus a 3 GB margin. It subtracts the other ttx builds in flight, and refuses with a record if the space is short. - A disk watchdog stops our own cold-build container gracefully below 1 GB free. - Keyed caches now live flat at /var/lib/models/tt-cache/<key>. They are registered in .ttx-registry.json with size, state, owner arm and last use. - `ttx cache ls [--du]` lists them. `ttx cache gc --keep-days N [--dry-run]` deletes only keyed caches ttx built. It never deletes one that the prod unit, a running container, a held slot or a build in flight references. - 6 new tests. A mutant without protection fails them. - Checked on cfx-llm2: `cache ls` and `gc --dry-run` both run, and gc would delete 0 GB. `reserve` refuses with rc 3 on box python. - Documented in the tt-card-slot and tt-bench skills and in DEV-LOOP.
ttx doctor: estimate kernel-pinned leaked host memory and advise a reboot
Some checks failed
tt-stack-ci / Report upstream drift (pull_request) Successful in 7s
Details
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 18s
Details
2a876797cf 
Before today, low MemAvailable on cfx-llm2 looked the same whether it was a
live process or pages the tenstorrent module had leaked after a bad exit. Only
the second one needs a reboot. Around 6 GB had leaked, MemAvailable sat at
3-4 GB, and the prod-vLLM dev boot tripped the ttx memory watchdog at 1369 MB
(driver log in this commit). After the reboot, 10.8 GB is available.

`ttx doctor` computes
  MemTotal - MemFree - (Buffers+Cached) - AnonPages - Hugetlb - Slab - PageTables.
At 3 GB or more (TTX_LEAK_REBOOT_MB), with no tt-metal container alive, it
prints REBOOT RECOMMENDED and exits 1. While containers are alive it defers,
because their pinned pages count too. It also reports cards, slots, disks and
this boot's kernel OOM and pin-failure counts. The up() memory refusal now
points to it. Tests: a healthy post-reboot meminfo (275 MB), a 6 GB leak, and
a configurable threshold.

The hardware records from the loop driver are included:
- TP=1 G2 bfp4 load, solo and paired on both cards with the swap.
- TP=2 G2 demo first run plus two edit->measure cycles.

Also: demo runs no longer write empty .server.log files, and cycle_s is only
recorded for dev cycles, because otherwise it included lock waits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ttx dev --warm: keep the weights on device across forward-code edits
Some checks failed
tt-stack-ci / Report upstream drift (pull_request) Successful in 9s
Details
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 42s
Details
ec52abb806 
The cold edit->measure cycle on the TP=2 G2 demo measured 220 s, and 177 s of
that is loading weights that a forward-path edit never touches.

A persistent warm_server.py inside the dev container keeps the mesh open and
the model resident. On an edit it classifies the change by AST
(warm_codediff.py):
- comment-only: no reload.
- forward function bodies: reload the affected modules in dependency order,
  swap the live objects' classes to the new code (the weights are instance
  attributes and stay put), release traces, clear the program cache.
- loader code, weight_mapping, or module/class-level statements: the server
  exits and ttx does a full reload, recording why.

Stale state is the obvious trap, so identity is part of the cycle.
`--verify cold` (the default) runs the same tree in a fresh process and
compares token ids. A mismatch is FAILED; skipping the check is UNVERIFIED.
Both are unquotable.

Tests cover the classifier and a real reload with class swap on a throwaway
package. The classifier was also run inside the image: over the full qwen36
package it marks _final_norm_decode, _lm_head and decode_step_paged as forward
and __init__ as loader, in under 1 s. bench/ttx/warm-measure.sh is the
hardware driver: c1-c6, including an output-changing edit and a loader-edit
fallback. It is queued behind MTP and tt-quant. `ttx down` stops the warm
server cleanly before stopping the container.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bench/runs: prod vLLM through ttx: smoke + an edit->measure cycle (restart = boot)
Some checks failed
tt-stack-ci / Report upstream drift (pull_request) Successful in 8s
Details
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 37s
Details
560cbfa84a 
Before the 2026-09-25 reboot, this step never reached ready: leaked pinned
memory drove MemAvailable to 1369 MB and the ttx watchdog stopped the boot.
After the reboot it runs end to end, under a mesh slot:
- identity passed (7 banners, served root = checkpoint);
- boot took 340.8 s;
- the smoke passed;
- at B=1 (2246 tokens in / 128 out), 1494 tok/s PP and 46.9 ms/token TG.

The edit->measure cycle took 430.6 s: 342.4 s is the graceful restart, which
is a full boot, and the rest is identity plus the smoke. That is the number
the vLLM path costs per edit today. The demo path costs about 220 s cold, and
warm mode targets it next.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Some checks failed
tt-stack-ci / Report upstream drift (pull_request) Successful in 8s
Details
tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 37s
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin ttx:ttx
git switch ttx
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
human-approved
Approved by @donach for T2-path changes. Required by .forgejo/workflows/ci.yml approve_gate.

No labels
human-approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack!70
No description provided.