feat(metrics): read what vLLM already knows, instead of inferring it #12

Merged

Grok merged 1 commit from feat/vllm-metrics into master 2026-09-18 10:05:22 +02:00

Conversation 0 Commits 1 Files changed 6 +998 -2

Grok commented 2026-09-18 10:05:21 +02:00

Owner

Copy link

Inference on the Tenstorrent host runs vllm + the tt-metal plugin, and that
endpoint publishes far more than the harness was reading. Four things vLLM
already knew and we were either estimating or calling unknown.

Everything below is verified against a live scrape from cfx-llm2 (two p150a,
Qwen3.8-27B-FP8), committed unedited as the test fixture.

1. Queueing has a stated reason

vllm:num_requests_waiting_by_reason splits into capacity and deferred.
They are indistinguishable from the client and mean opposite things: capacity
is the level being past what the pool holds at once, so added load is becoming
queue instead of throughput. That is precisely the ceiling this sweep exists to
find, and it is now read rather than inferred from a bend in a curve. Measured
mid-run: {capacity: 2.0, deferred: 0.0}. The report prints it above the
numbers.

2. Computed prefill is published

vllm:request_prefill_kv_computed_tokens is the KV actually pushed through
attention — the quantity our nominal/computed bracket estimates from outside.
Over vllm:request_prefill_time_seconds, both histograms over completed
requests and therefore the same population: 802,418 tokens / 294.7 s ≈ 2,723
tok/s per prefilling stream.

Recorded explicitly as per-stream, not aggregate: concurrent prefill spans
overlap, so the summed time exceeds the wall time the box spent prefilling.
For the same reason these are not aliased onto llama.cpp's prompt_seconds,
which is wall time — that would silently understate a concurrent engine.

3. A zero cached fraction is no longer an unknown

vllm:cache_config_info carries enable_prefix_caching="False" on this
deployment. Nothing could have been served from a prefix cache, so the nominal
prefill rate is the computed one. Reporting "cached fraction unknown" there
understated what the engine had already told us.

4. A FLOPs counter of zero is not zero work

The plugin publishes vllm:estimated_flops_per_gpu_total and leaves it at
0.0. Dividing that by the joules #10/#11 now measure would report an
accelerator that did no floating-point work. It is recorded as unimplemented
and withheld from the efficiency figures.

Fixture

An unedited /metrics scrape from that host, so a plugin or vLLM upgrade that
renames a series fails here rather than quietly reducing the harness to
client-side estimates. It also confirms the KV pool reader against real output:
16,392 blocks × 64 = 1,049,088 tokens.

8 new tests, 309 total. nix flake check green.

🤖 Generated with Claude Code

Inference on the Tenstorrent host runs **vllm + the tt-metal plugin**, and that endpoint publishes far more than the harness was reading. Four things vLLM already knew and we were either estimating or calling unknown. Everything below is verified against a live scrape from cfx-llm2 (two p150a, Qwen3.8-27B-FP8), committed unedited as the test fixture. ## 1. Queueing has a stated reason `vllm:num_requests_waiting_by_reason` splits into `capacity` and `deferred`. They are indistinguishable from the client and mean opposite things: **capacity is the level being past what the pool holds at once**, so added load is becoming queue instead of throughput. That is precisely the ceiling this sweep exists to find, and it is now read rather than inferred from a bend in a curve. Measured mid-run: `{capacity: 2.0, deferred: 0.0}`. The report prints it above the numbers. ## 2. Computed prefill is published `vllm:request_prefill_kv_computed_tokens` is the KV actually pushed through attention — the quantity our nominal/computed bracket estimates from outside. Over `vllm:request_prefill_time_seconds`, both histograms over *completed* requests and therefore the same population: **802,418 tokens / 294.7 s ≈ 2,723 tok/s per prefilling stream**. Recorded explicitly as per-stream, not aggregate: concurrent prefill spans overlap, so the summed time exceeds the wall time the box spent prefilling. For the same reason these are **not** aliased onto llama.cpp's `prompt_seconds`, which is wall time — that would silently understate a concurrent engine. ## 3. A zero cached fraction is no longer an unknown `vllm:cache_config_info` carries `enable_prefix_caching="False"` on this deployment. Nothing could have been served from a prefix cache, so the nominal prefill rate **is** the computed one. Reporting "cached fraction unknown" there understated what the engine had already told us. ## 4. A FLOPs counter of zero is not zero work The plugin publishes `vllm:estimated_flops_per_gpu_total` and leaves it at `0.0`. Dividing that by the joules #10/#11 now measure would report an accelerator that did no floating-point work. It is recorded as *unimplemented* and withheld from the efficiency figures. ## Fixture An unedited `/metrics` scrape from that host, so a plugin or vLLM upgrade that renames a series fails here rather than quietly reducing the harness to client-side estimates. It also confirms the KV pool reader against real output: **16,392 blocks × 64 = 1,049,088 tokens**. 8 new tests, 309 total. `nix flake check` green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Grok added 1 commit 2026-09-18 10:05:21 +02:00

feat(metrics): read what vLLM already knows, instead of inferring it ... eb7c25e2b6

Inference on the Tenstorrent host runs vllm + the tt-metal plugin, and
that endpoint publishes far more than the harness was reading. Four
things it already knew and we were estimating or calling unknown.

Queueing has a stated reason. `num_requests_waiting_by_reason` splits
into `capacity` and `deferred`, which are indistinguishable from the
client and mean opposite things: capacity is the level being past what
the pool holds at once, so added load is becoming queue instead of
throughput. That is the ceiling the sweep exists to find, and it is now
read rather than inferred from a bend in a curve. Measured on cfx-llm2
mid-run: 2 waiting on capacity, 0 deferred.

Computed prefill is published. `request_prefill_kv_computed_tokens` is
the KV actually pushed through attention - the quantity our nominal and
computed bracket estimates from outside. Divided by
`request_prefill_time_seconds`, both histograms over completed requests
and therefore the same population, it gives prefill speed per prefilling
stream: 802,418 tokens over 294.7 s, about 2,723 tok/s. Recorded as
per-stream and not aggregate, because concurrent prefill spans overlap
so the summed time exceeds the wall time the box spent prefilling. For
the same reason these are deliberately not aliased onto llama.cpp's
`prompt_seconds`, which is wall time.

A zero cached fraction is no longer an unknown. `cache_config_info` says
`enable_prefix_caching="False"` on this deployment, so nothing could
have been served from a prefix cache and the nominal prefill rate IS the
computed one. Reporting "cached fraction unknown" there understated what
the engine had already told us.

A FLOPs counter of zero is not zero work. The plugin publishes
`estimated_flops_per_gpu_total` and leaves it at 0.0; that is an
unimplemented estimate, and dividing it by measured joules would report
an accelerator that did no floating-point work. It is recorded as
unimplemented and withheld from the efficiency figures.

The fixture is an unedited scrape from that host, so a plugin or vLLM
upgrade that renames a series fails here rather than quietly reducing
the harness to client-side estimates. It also confirms the KV pool
reader on real output: 16,392 blocks x 64 = 1,049,088 tokens.

8 tests, 309 total, nix flake check green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Grok merged commit 26206fb5d6 into master 2026-09-18 10:05:22 +02:00

Grok referenced this pull request from a commit 2026-09-18 10:05:22 +02:00

Merge pull request 'feat(metrics): read what vLLM already knows, instead of inferring it' (#12) from feat/vllm-metrics into master

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

donach/inference-harness!12

No description provided.