donach/inference-harness

Watch

Fork 0

Confirm an arm from the server, and check the host before the run #13

Open

Grok wants to merge 1 commit from feat/tenstorrent-and-arm-identification into master

Grok commented

2026-09-18 10:06:42 +02:00

Owner

What a Tenstorrent Blackhole host needs from this harness that it did not yet
have — plus the one lesson that is not Tenstorrent-specific at all and matters
most.

Everything here was checked against infbench's actual code first. The survey in
docs/FIELD-NOTES.md had already identified several of these as "to port";
this implements (B) and (T), extends (N) to a runtime guard, and leaves the
rest of that list untouched. Several items on the original gap list turned out
to be already handled, in some cases better than in tt-stack — listed at the
bottom rather than duplicated under a new name.

Backed by a measured incident (citations are files in `tt-stack`)

Arm self-identification — src/infbench/arm.py, wired into stage 1.
A declared arm is confirmed against the server's own output (an engine log, the
*_info metrics gauges, /v1/models) and fails closed when its banner is
absent. Incidents: vLLM's GPU KV cache size is a token count derived from
max_model_len x max_num_seqs and is identical between bf16 and bf8 KV, so
every bf8 result before patches/tt-metal/0016 had no runtime evidence of its
arm (CLAUDE.md, Gate A entry); a control that reused a patched tree inherited
a different flag default and inflated a delta from +4.5% to +20.2%
(bench/ab/argmax-16k-ab.sh, which now fails closed on a missing banner); and
twice in one session a flag failed to reach a container through an explicit env
list. forbids catches the reverse case — a control that became the treatment —
which is invisible from the treatment side alone.

Host memory floor — preflight.memory_check. Measured: with zero
containers, zero device processes and the module at refcount 0, rmmod +
modprobe reclaimed 0 MB while a reboot reclaimed 4.2 GiB
(bench/runs/host-pinned-leak-20260914T063000Z.jsonl). Two boots attempted
below ~3 GB available that day hard froze the host. The failure message says
a driver reload is not the fix. host_mem_before / host_mem_after are both
recorded, because a run that ends with less than it started is the signature of
pinned pages outliving a killed process.

Exclusive host lock — preflight.HostLock. That host runs exactly one
tt-metal process, and it is not per card: /dev/tenstorrent/1 does not
avoid "another process is already holding the sysmem NOC address space". The
lock is mkdir-based with owner and pid inside and never steals a stale
one. The bash version's trap — a trap handler returns to where it was
interrupted unless it exits, so the lock was released on SIGTERM and
containers launched unlocked — is recorded in the module docstring.

Foreign-container check + recovery text — tenstorrent.py. A job we did
not start holds no lock of ours. And docker rm -f on a live tt-metal
container wedges the cards so that the next open reads as a ttnn/firmware
version mismatch — a known-good image failed identically right afterwards. The
harness never resets a board and never stops a container; it prints
docker stop -t 90 and sudo tt-smi -r, for the same reason tt-flash is not
packaged in tt-stack.

Tokenizer-based prompt sizing — calibrate. Now prefers the server's own
/tokenize (same chat template as the completion path, exact, no generation)
and falls back to the existing two-probe solve. A remembered chars-per-token
ratio has been wrong twice on this project, most recently 3.26 assumed against
a measured 4.55.

Time-to-last-token per stream — stage 4. Makespan was already the
throughput denominator; the distribution was not reported. Owner directive
2026-09-13 makes TTLT first-class, and the reason is measured: at B=8/8k decode
improved 14.1% while makespan improved 11.0%, because prefill was 74-79% of
wall (bench/runs/sweep-32k-20260914T142154Z.jsonl).

Proposals — the principle is measured, this shape of it is not

src/infbench/sensitivity.py. The principle is paid for four times over
in tt-stack: a text_sha256 identical under every configuration; a stall
counter whose window opened only after the last prefill; a py-spy classifier
that bucketed by the whole stack string and returned the opposite verdict; a
cache bench that could not tell a cache hit from intra-batch head sharing
(docs/INSTRUMENT-INTEGRITY.md, already cited under (N) in FIELD-NOTES). The
API here — a negative-control arm and a sensitivity arm, with a 5% default
separation floor and a WITHHELD verdict — is my proposal, not a ported design.

The 5% floor and the 4 GiB memory floor are judgement calls. The floor
under the first is the 0.33% run-to-run noise measured on a serving host; 5% is
an order of magnitude above it and otherwise unvalidated. The second is set
above the ~3 GB at which the host actually froze, not at a measured safe point.

The decode-depth sensitivity arm in stage 4 applies the guard to an
existing ladder. A flat reading is flagged rather than published as a
depth-flat engine; the rows already carry depth_drift_pct and
prefill_leaked_into_decode, which are the two explanations to check first.
This has not been run against real hardware.

What infbench already handled — and in places better than we do

Checked before implementing; none of it was duplicated.

Readiness gating is already layered and already engine-shaped:
/health → /metrics → /v1/models → a real generated token, with
ready_timeout_s = 600 and a comment naming the ~510 s Tenstorrent load. The
gateway-returns-400-for-a-minute case that voided two of our runs is caught
by the generation rung.
PerfRecord refuses to exist without passing gates and an
ArrivalModel. The second is something we have no equivalent of at all,
and it is a real hole on our side: our A/B records do not say how load
arrived.
Metrics come from the server's usage block, with tps_source recording
when they fell back to counted SSE chunks — the failure that overstated our
decode ~3x once is already labelled here.
Batch prefill is already max(ttft), not the mean, and the TTFT
staircase is recorded in launch order precisely so a serialised-prefill
pattern is visible.
Cold vs warm is structural: corpus_seed, grow_from_session,
cold_claim_valid against the engine's own cached fraction, and a nominal /
computed prefill split. Stricter than our cache bench.
Repeats are interleaved and a >15% spread prints UNSTABLE; the reported
record is the median repeat, never a per-field average. We should adopt
this framing.
Run identity already includes an overlay_hash for bind-mounted patched
files, which is exactly the hole vllm-tt-serve.sh opens in an image digest.
Batch-width-changes-greedy-output is already in its trap list, with the
sharper distinction we lacked: divergent phrasing with the answer intact is
a different finding from a wrong answer.

Notes for review

Tests: 46 new, nix flake check green (pytest + ruff + gitleaks). Every new
check has a fixture that makes it fail, per the (N) convention — including
the empty-evidence case, which is the one that matters.
[arm] and [preflight] are both inert unless configured; default behaviour
is unchanged except for the two added stage-4 fields and one report line.
One caveat: preferring /tokenize can shift overhead_tokens by a few
tokens against the generation-probe path. Calibration.key rounds overhead to
the nearest 64, so most baselines will not churn — but a baseline sitting on a
boundary could be orphaned, which is the intended refusal rather than a
silent comparison.
Nothing was run against hardware; the cards are busy. Host-side unit tests
only.

🤖 Generated with Claude Code

What a Tenstorrent Blackhole host needs from this harness that it did not yet have — plus the one lesson that is not Tenstorrent-specific at all and matters most. Everything here was checked against infbench's actual code first. The survey in `docs/FIELD-NOTES.md` had already identified several of these as "to port"; this implements (B) and (T), extends (N) to a runtime guard, and leaves the rest of that list untouched. Several items on the original gap list turned out to be **already handled, in some cases better than in tt-stack** — listed at the bottom rather than duplicated under a new name. ## Backed by a measured incident (citations are files in `tt-stack`) **Arm self-identification — `src/infbench/arm.py`, wired into stage 1.** A declared arm is confirmed against the server's own output (an engine log, the `*_info` metrics gauges, `/v1/models`) and **fails closed** when its banner is absent. Incidents: vLLM's `GPU KV cache size` is a token count derived from `max_model_len x max_num_seqs` and is identical between bf16 and bf8 KV, so every bf8 result before `patches/tt-metal/0016` had no runtime evidence of its arm (`CLAUDE.md`, Gate A entry); a control that reused a patched tree inherited a different flag default and inflated a delta from +4.5% to +20.2% (`bench/ab/argmax-16k-ab.sh`, which now fails closed on a missing banner); and twice in one session a flag failed to reach a container through an explicit env list. `forbids` catches the reverse case — a control that became the treatment — which is invisible from the treatment side alone. **Host memory floor — `preflight.memory_check`.** Measured: with zero containers, zero device processes and the module at refcount 0, `rmmod` + `modprobe` reclaimed **0 MB** while a reboot reclaimed **4.2 GiB** (`bench/runs/host-pinned-leak-20260914T063000Z.jsonl`). Two boots attempted below ~3 GB available that day **hard froze the host**. The failure message says a driver reload is not the fix. `host_mem_before` / `host_mem_after` are both recorded, because a run that ends with less than it started is the signature of pinned pages outliving a killed process. **Exclusive host lock — `preflight.HostLock`.** That host runs exactly one tt-metal process, and it is **not per card**: `/dev/tenstorrent/1` does not avoid `"another process is already holding the sysmem NOC address space"`. The lock is `mkdir`-based with owner and pid inside and **never steals** a stale one. The bash version's trap — *a trap handler returns to where it was interrupted unless it exits*, so the lock was released on SIGTERM and containers launched unlocked — is recorded in the module docstring. **Foreign-container check + recovery text — `tenstorrent.py`.** A job we did not start holds no lock of ours. And `docker rm -f` on a live tt-metal container wedges the cards so that the next open reads as a ttnn/firmware version mismatch — a known-good image failed identically right afterwards. The harness **never** resets a board and never stops a container; it prints `docker stop -t 90` and `sudo tt-smi -r`, for the same reason `tt-flash` is not packaged in tt-stack. **Tokenizer-based prompt sizing — `calibrate`.** Now prefers the server's own `/tokenize` (same chat template as the completion path, exact, no generation) and falls back to the existing two-probe solve. A remembered chars-per-token ratio has been wrong twice on this project, most recently 3.26 assumed against a measured 4.55. **Time-to-last-token per stream — stage 4.** Makespan was already the throughput denominator; the distribution was not reported. Owner directive 2026-09-13 makes TTLT first-class, and the reason is measured: at B=8/8k decode improved 14.1% while makespan improved 11.0%, because prefill was 74-79% of wall (`bench/runs/sweep-32k-20260914T142154Z.jsonl`). ## Proposals — the principle is measured, this shape of it is not **`src/infbench/sensitivity.py`.** The *principle* is paid for four times over in tt-stack: a `text_sha256` identical under every configuration; a stall counter whose window opened only after the last prefill; a py-spy classifier that bucketed by the whole stack string and returned the opposite verdict; a cache bench that could not tell a cache hit from intra-batch head sharing (`docs/INSTRUMENT-INTEGRITY.md`, already cited under (N) in FIELD-NOTES). The API here — a negative-control arm and a sensitivity arm, with a 5% default separation floor and a WITHHELD verdict — is my proposal, not a ported design. **The 5% floor and the 4 GiB memory floor are judgement calls.** The floor under the first is the 0.33% run-to-run noise measured on a serving host; 5% is an order of magnitude above it and otherwise unvalidated. The second is set above the ~3 GB at which the host actually froze, not at a measured safe point. **The decode-depth sensitivity arm in stage 4** applies the guard to an existing ladder. A flat reading is flagged rather than published as a depth-flat engine; the rows already carry `depth_drift_pct` and `prefill_leaked_into_decode`, which are the two explanations to check first. This has not been run against real hardware. ## What infbench already handled — and in places better than we do Checked before implementing; none of it was duplicated. - **Readiness gating** is already layered and already engine-shaped: `/health` → `/metrics` → `/v1/models` → *a real generated token*, with `ready_timeout_s = 600` and a comment naming the ~510 s Tenstorrent load. The gateway-returns-400-for-a-minute case that voided two of our runs is caught by the generation rung. - **`PerfRecord` refuses to exist without passing gates *and* an `ArrivalModel`.** The second is something we have no equivalent of at all, and it is a real hole on our side: our A/B records do not say how load arrived. - **Metrics come from the server's `usage` block**, with `tps_source` recording when they fell back to counted SSE chunks — the failure that overstated our decode ~3x once is already labelled here. - **Batch prefill is already `max(ttft)`, not the mean**, and the TTFT staircase is recorded in launch order precisely so a serialised-prefill pattern is visible. - **Cold vs warm** is structural: `corpus_seed`, `grow_from_session`, `cold_claim_valid` against the engine's own cached fraction, and a nominal / computed prefill split. Stricter than our cache bench. - **Repeats are interleaved and a >15% spread prints UNSTABLE**; the reported record is the *median repeat*, never a per-field average. We should adopt this framing. - **Run identity already includes an `overlay_hash`** for bind-mounted patched files, which is exactly the hole `vllm-tt-serve.sh` opens in an image digest. - **Batch-width-changes-greedy-output** is already in its trap list, with the sharper distinction we lacked: divergent *phrasing* with the answer intact is a different finding from a wrong answer. ## Notes for review - Tests: 46 new, `nix flake check` green (pytest + ruff + gitleaks). Every new check has a fixture that makes it fail, per the (N) convention — including the empty-evidence case, which is the one that matters. - `[arm]` and `[preflight]` are both inert unless configured; default behaviour is unchanged except for the two added stage-4 fields and one report line. - One caveat: preferring `/tokenize` can shift `overhead_tokens` by a few tokens against the generation-probe path. `Calibration.key` rounds overhead to the nearest 64, so most baselines will not churn — but a baseline sitting on a boundary could be orphaned, which is the intended refusal rather than a silent comparison. - Nothing was run against hardware; the cards are busy. Host-side unit tests only. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Grok added 1 commit

2026-09-18 10:06:42 +02:00

Confirm an arm from the server, not from the launcher's intent

ci / check (push) Has been cancelled

Details

ci / check (pull_request) Has been cancelled

Details

ci / site (push) Has been cancelled

Details

ci / site (pull_request) Has been cancelled

Details

378f40b01d

We believed a run's arm was known because the launcher set the flag. It is
not. vLLM's "GPU KV cache size" line is a token count derived from
max_model_len x max_num_seqs and is byte-identical between a bf16 and a bf8 KV
cache, so an entire family of bf8 results in tt-stack carried no runtime
evidence of which arm produced them; the fix there was a print-only patch whose
only purpose was to make the arm observable. Twice in one session a flag failed
to reach a container through an explicit environment list and the run silently
measured the control, and once a control inherited a different default from a
reused source tree and inflated a delta from +4.5% to +20.2%.

So `infbench.arm` separates a declared arm from a confirmed one and fails
closed: a declared arm whose banner never appears in the server's own output -
an engine log, the *_info metrics gauges, the model list - is a FAIL, because a
warning leaves the number in the record and the number is what gets quoted.
`forbids` carries the control's banner, which is the only way to see an arm
that silently became the other one.

Three more things follow the same logic and land with it.

`infbench.sensitivity` gives instrument integrity a runtime shape. An
instrument structurally incapable of producing a different number returns the
comfortable answer forever, and that shape has been built at least four times
in one project. Stage 4's decode-depth ladder now carries a sensitivity arm: a
ladder that reads flat at every concurrency says it has not demonstrated
sensitivity rather than publishing a depth-flat engine.

`infbench.preflight` (vendor-neutral) and `infbench.tenstorrent` check the host
before the endpoint. A MemAvailable floor, because two boots below ~3 GB hard
froze the measured host and a module reload reclaimed 0 MB where a reboot
reclaimed 4.2 GiB; an exclusive host lock that never steals, because that
device admits one process host-wide and not one per card; a foreign-container
check, because a job we did not start holds no lock of ours. Host memory is
recorded on both sides of the stage. Nothing here resets a board or stops a
container - it prints the command.

Calibration now prefers the server's own /tokenize, which applies the same chat
template the completion path does, and falls back to the generation probes
where that route does not exist. A remembered chars-per-token ratio has been
wrong twice on one project, 3.26 assumed against a measured 4.55, and each time
the ladder measured a depth nobody asked for.

Stage 4 also reports time-to-last-token per stream. The makespan was already
the throughput denominator; what was missing is the distribution, and a batch
that straggles is invisible in both the makespan and the decode rate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci / check (push) Has been cancelled

Details

ci / check (pull_request) Has been cancelled

Details

ci / site (push) Has been cancelled

Details

ci / site (pull_request) Has been cancelled

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/tenstorrent-and-arm-identification:feat/tenstorrent-and-arm-identification

git switch feat/tenstorrent-and-arm-identification

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff feat/tenstorrent-and-arm-identification

git switch feat/tenstorrent-and-arm-identification

git rebase master

git switch master

git merge --ff-only feat/tenstorrent-and-arm-identification

git switch feat/tenstorrent-and-arm-identification

git rebase master

git switch master

git merge --no-ff feat/tenstorrent-and-arm-identification

git switch master

git merge --squash feat/tenstorrent-and-arm-identification

git switch master

git merge --ff-only feat/tenstorrent-and-arm-identification

git switch master

git merge feat/tenstorrent-and-arm-identification

git push origin master

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

donach/inference-harness!13

No description provided.

Rows
Columns

Confirm an arm from the server, and check the host before the run #13

Backed by a measured incident (citations are files in tt-stack)

Proposals — the principle is measured, this shape of it is not

What infbench already handled — and in places better than we do

Notes for review

Checkout

Merge

Backed by a measured incident (citations are files in `tt-stack`)