infbench

A correctness-gated benchmarking harness for self-hosted LLM inference.

Engine- and vendor-neutral: anything that speaks the OpenAI chat-completions API (vLLM, SGLang, TensorRT-LLM, llama.cpp, vllm-tt-plugin) plus static inspection of checkpoints on disk.

Why it is built this way

Two invariants, both of them corrections of real, expensive mistakes.

A performance number that carries no correctness verdict is unrepresentable. Not discouraged — unrepresentable. PerfRecord cannot be constructed without a satisfied GateSet, the serializer refuses, and the reporter refuses. The mistake this prevents is quoting throughput for an endpoint that was emitting pure garbage.

Stages run cheapest-first and abort on the first failure. A thirty-second tensor inventory belongs before a 190 GB download, not after it.

The five stages

Stage	Cost	What it proves
0 static	seconds	The checkpoint contains what the quant scheme it declares requires. No GPU, no server; runs off the safetensors index, so it works before the weights land.
1 hardware	seconds	These compiled kernels can actually execute on this device, and the server generates at all.
2 smoke	~30 s	One request, known answer, temperature 0 — across every output channel, with degeneracy detectors and a parsed tool call.
3 correctness	minutes	Long agentic horizon, canary retrieval, correctness under concurrency, cold/warm cache equality, batch invariance, determinism, needle-in-haystack.
4 performance	longer	TTFT / prefill / decode / end-to-end, reported separately, over a concurrency sweep, carrying stage 0–3 verdicts.

Quick start

nix run .#stage0 -- --checkpoint /path/to/checkpoint     # before you download the rest
nix run .#stage2 -- --target http://127.0.0.1:8000/v1    # after every restart
nix run .#run   -- --target http://127.0.0.1:8000/v1 --checkpoint /path/to/checkpoint
nix run .#matrix -- 'bench/runs/*.jsonl'

Or without Nix:

uv venv && uv pip install -e '.[dev]'
.venv/bin/infbench run -c example.toml

Exit codes: 0 healthy, 1 warnings only (something was skipped), 2 hard failure. Stage 2 is meant to run automatically after every restart, so its exit code has to mean something to a shell.

What to run when

Keyed to the trigger, because that is how people reach for it.

Trigger	Run
New checkpoint, known runtime	stages 0, 2, 3
New runtime or image, known checkpoint	stages 1, 2, 3 — and record a full matrix row
Version bump	stages 1, 2, then re-baseline stage 4
Config change	stage 2, stage 3 concurrency, stage 4
Tuning the serving config	baseline before, baseline after, compare
Pre-production	everything, plus a sustained soak

Did it get faster or slower?

A sweep answers how does this deployment behave. A baseline answers the narrower, more useful question: did it change.

infbench baseline -c suite.toml --out bench/runs/before.jsonl
#   ... change something in the serving config ...
infbench baseline -c suite.toml --out bench/runs/after.jsonl
infbench compare bench/runs/before.jsonl bench/runs/after.jsonl

A frozen, hashed workload at 8k/512, 16k/1k and 32k/2k, serial, five repeats, measuring cold prefill, warm (cache-hit) prefill, and token generation at each depth — then fitting time-vs-context so a regression can be attributed to fixed overhead or to per-context-token cost rather than just observed.

compare refuses more than it reports: a different workload hash, failed gates on either side, or a different accelerator all make the subtraction meaningless and are refused rather than differenced. Deltas inside the noise the repeats themselves showed are reported as no detectable change, which is a finding. Measuring and comparing is all this repo does; deciding what to change belongs wherever the serving config does. See docs/BASELINE.md.

See docs/GUIDE.md for the detail and docs/FAILURE-TAXONOMY.md for reading a failure before diagnosing it. To benchmark one host end to end with no prior context, follow docs/RUNBOOK.md.

How load arrives is part of the measurement

Two runs of this harness disagreed about first-token latency by 4x on the same GPU, model and prompts, because one launched every stream at t=0 and the other let sessions think between turns. Both records were readable; neither said which. So PerfRecord requires an ArrivalModel exactly as it requires passing gates, and UnmodelledArrivalError sits beside UngatedResultError.

Concurrency is not an input. It is an output of how many sessions exist and how often each one speaks — so infbench.population sweeps sessions and measures the in-flight distribution, then checks it against the time-weighted histogram in the workload profile. See docs/ARRIVAL.md.

The workload the defaults describe

The defaults are measured, not chosen. See docs/WORKLOAD.md: 18,834 real agentic-coding requests say the median prompt is 92,888 tokens, that 97.3% of prefill tokens are cache reads because each turn's context contains the previous turn's verbatim, and that real in-flight concurrency is 1–2 for 93% of active time, peaking at 16.

The first version of this harness swept 1,024-token prompts at concurrency up to 64. It measured a machine nobody runs.

Two consequences visible in every stage-4 result:

• Decode rate is stamped with its KV depth, and a depth ladder is swept separately. Decode throughput falls by roughly half between a shallow prompt and a 32k one at the same concurrency, so a decode number without its depth is not a number.
• Prefill is reported twice: the nominal rate, and the rate for the tokens the engine actually computed. A workload that is 97% cache hits will quote a nominal figure more than 10× the real one.

Which throughput?

Five different numbers get called throughput and on a real endpoint they differ by more than 10x. docs/TRUE-THROUGHPUT.md is the field guide: ask the engine's own timers before estimating from the outside, report goodput rather than emitted tokens, and never quote one without saying which it is.

How many people can share one box

Every sweep level reports tokens/hour and an SLO verdict, and the sweep reports the highest concurrency that still meets it:

  interactive capacity  at ~92,888 tok of context
    good     up to   4 concurrent user(s), 1.31 M tok/h there
    average  up to   8 concurrent user(s), 2.04 M tok/h there

The primary metric is goodput at an SLO — conjunctive, per request, bound at the p99 — which is what MLPerf Inference, DistServe and NVIDIA's GenAI-Perf all use. A 0-100 interactivity score is published beside it as a skimming aid and is explicitly not the ranking key. See docs/INTERACTIVITY.md, which also records that our bands are far stricter than MLPerf's.

Field notes from our other benchmarks

docs/FIELD-NOTES.md collects what the benchmarking setups in nixos-llm and tt-stack already knew and this harness did not — with the incident behind each one. Two of them were live defects here and are fixed; the rest are a prioritised porting list.

The published table

nix run .#site -- --serve     # rebuild from bench/runs/ and serve it

One self-contained HTML page, re-derived in full from the recorded results on every build, published by CI to a branch Forgejo serves. Colleagues get a link rather than a JSONL file. Ungated rows are published with their caveat rather than dropped — see docs/PUBLISHING.md.

Results

One JSON object per run, appended to bench/runs/*.jsonl, keyed by run identity: (engine, image digest, checkpoint revision, accelerator arch, engine config hash, cache state). Deliberately not the pip version triple — two images can report identical vllm/flashinfer/transformers versions and differ only in compiled kernels, and that difference is the whole ballgame.

infbench matrix renders those into the (image × checkpoint) pass/fail table. That table is the artifact that makes the next version bump safe, and it is the thing nobody ever writes down.

Results are committed, so they are redacted on write: endpoints reduced to scheme + opaque host + port, hostnames to stable non-reversible labels, environment captured from an allowlist rather than dumped.

Scope

The five-stage skeleton is stable. The per-stage checks are a growing library — they were drawn from a handful of engines and will need broadening as more are added; TensorRT-LLM, SGLang and llama.cpp each have failure modes vLLM does not. Adding a check is the normal way this repo grows.

Apache-2.0.