donach/inference-harness
Watch
1
0
Fork
You've already forked inference-harness
 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

Add fastbench 3-tier validation workflow for rapid lever iteration #14

Open
Grok wants to merge 1 commit from fastbench-3tier into master
pull from: fastbench-3tier
merge into: donach:master
Conversation 0 Commits 1 Files changed 4 +246
Grok commented 2026-09-19 13:33:36 +02:00
Owner
Copy link

Summary

Adds the 'fastbench' 3-tier validation subcommand and configuration options to infbench to dramatically accelerate the serving throughput iteration loop while preserving statistical confidence and correctness gating.

The 3-Tier Workflow:

  1. Tier 1 (Smoke Gate): Stage 2 fast functional smoke test (known answers for capital, arithmetic, and tool calling; ~3.7s).
  2. Tier 2 (Targeted Correctness): Stage 3 concurrency correctness under load (concurrency=16, 32 requests) and determinism. Bypasses long needle-in-haystack scans by default to finish in ~35-40s (can be re-enabled with --with-haystack).
  3. Tier 3 (Fast Perf): Fast screening profile (fastbench-v1: 1k prompt / 256 gen tokens, concurrency [1, 8], repeats 2). Delivers high-precision decode latency (TG ms/tok) and prompt throughput (PP tok/s) in ~45s without paying 15-20 minutes of prefill serialization per sweep.

CLI Usage:

# Run all 3 tiers sequentially (gating Tier 3 on Tier 1 & 2 passes)
infbench fastbench -c scratchpad/my-suite.toml

# Run only Tier 1 (smoke) or Tier 2 (concurrency correctness)
infbench fastbench -c scratchpad/my-suite.toml --tier 1
infbench fastbench -c scratchpad/my-suite.toml --tier 2

# Overrides for rapid screening
infbench fastbench --tier 3 --concurrency 1,4,8 --prompt-tokens 2048 --max-tokens 512

Verification:

  • Added tests/test_fastbench.py testing profile registration, config defaults, and CLI parser handling.
  • Full pytest suite passes cleanly (313/313 tests passed).
## Summary Adds the 'fastbench' 3-tier validation subcommand and configuration options to infbench to dramatically accelerate the serving throughput iteration loop while preserving statistical confidence and correctness gating. ### The 3-Tier Workflow: 1. **Tier 1 (Smoke Gate)**: Stage 2 fast functional smoke test (known answers for capital, arithmetic, and tool calling; ~3.7s). 2. **Tier 2 (Targeted Correctness)**: Stage 3 concurrency correctness under load (concurrency=16, 32 requests) and determinism. Bypasses long needle-in-haystack scans by default to finish in ~35-40s (can be re-enabled with `--with-haystack`). 3. **Tier 3 (Fast Perf)**: Fast screening profile (`fastbench-v1`: 1k prompt / 256 gen tokens, concurrency [1, 8], repeats 2). Delivers high-precision decode latency (TG ms/tok) and prompt throughput (PP tok/s) in ~45s without paying 15-20 minutes of prefill serialization per sweep. ### CLI Usage: ```bash # Run all 3 tiers sequentially (gating Tier 3 on Tier 1 & 2 passes) infbench fastbench -c scratchpad/my-suite.toml # Run only Tier 1 (smoke) or Tier 2 (concurrency correctness) infbench fastbench -c scratchpad/my-suite.toml --tier 1 infbench fastbench -c scratchpad/my-suite.toml --tier 2 # Overrides for rapid screening infbench fastbench --tier 3 --concurrency 1,4,8 --prompt-tokens 2048 --max-tokens 512 ``` ### Verification: - Added `tests/test_fastbench.py` testing profile registration, config defaults, and CLI parser handling. - Full pytest suite passes cleanly (313/313 tests passed).
Add fastbench 3-tier validation command and profile
Some checks failed
ci / check (push) Has been cancelled
Details
ci / check (pull_request) Has been cancelled
Details
ci / site (push) Has been cancelled
Details
ci / site (pull_request) Has been cancelled
Details
20c332d8e2 
Implements the fast 3-tier validation framework:
- Tier 1: Fast functional smoke gate (Stage 2)
- Tier 2: Targeted concurrency correctness (Stage 3 with concurrency=16
  check, bypassing long needle-in-haystack scans by default for 35-40s turnaround)
- Tier 3: Fast performance validation (fastbench-v1 profile: 1k/256 and
  concurrency [1, 8] with 2 repeats, yielding TG ms/token and PP tok/s in <45s)

Adds CLI subcommand 'infbench fastbench' with --tier {1,2,3,all}, --with-haystack,
and concurrency/prompt-length overrides.
Some checks failed
ci / check (push) Has been cancelled
Details
ci / check (pull_request) Has been cancelled
Details
ci / site (push) Has been cancelled
Details
ci / site (pull_request) Has been cancelled
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fastbench-3tier:fastbench-3tier
git switch fastbench-3tier

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff fastbench-3tier
git switch fastbench-3tier
git rebase master
git switch master
git merge --ff-only fastbench-3tier
git switch fastbench-3tier
git rebase master
git switch master
git merge --no-ff fastbench-3tier
git switch master
git merge --squash fastbench-3tier
git switch master
git merge --ff-only fastbench-3tier
git switch master
git merge fastbench-3tier
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
donach/inference-harness!14
No description provided.