ThinkingCap FP8 baseline on TT #65

New issue

Open

opened 2026-09-24 14:59:27 +02:00 by Grok · 0 comments

Grok commented

2026-09-24 14:59:27 +02:00

Owner

This is the no-MTP baseline for #63 (MTP on ThinkingCap) and the model switch (ledger 0aaaaaaaaaaaaaaaaaa). Related: #62. All numbers are MEASURED on cfx-llm2 (2x P150a, TP=2) on 2026-09-24. Branch: thinkingcap-baseline (855ef82). Summary record: bench/runs/tc-baseline-summary-20260924.jsonl.

Setup

Both checkpoints ran on one source tree and profile. Only the checkpoint and its tensor cache changed.
- Profile: scripts/vllm-tt-serve.sh prod-argmax (bf8 KV, device argmax, no prefix caching, trace_mode=all).
- Server: CTX=34816 SEQS=8, pool 1,048,576, vllm-tt:k2.
Sources: the deployed tenstorrent-serving nix-store set plus the hand-ported 0019 slot fix. The tool parser is qwen3_xml.
ThinkingCap got its own cache (tt-cache/thinkingcap-e55ba7f6-prod), built with COLD=1. The MTP harness's cache lacked the bf4 down_proj tensors this profile needs, so it was not reused.
Every boot's server log was checked for these banners: QWEN_SDPA_BF8 engaged (BFLOAT8_B), QWEN36_DEVICE_ARGMAX engaged, QWEN36_PREFILLED_SLOTS_FIX engaged, GPU KV cache size: 1,049,088. The /v1/models root matched the checkpoint in each case.
Harness: infbench stages 0-4, one gated suite per rung (bench/infbench/tc-baseline/*.toml).
- Load: synchronised burst of B = 4 and 8, 1024 max output tokens, 2 repeats.
- Quality: bench/gsm8k.py with thinking on, n=60, B=8, max_tokens 8192.

Correctness

The deployed production argmax tree fails stage 3 without 0019. The concurrency check failed 1 of 32 requests with expected 391, got 33. That is the same slot-reuse signature 0019 fixed on 09-19. The deployed unit carries 0014 but not 0019. Record: tc-baseline-thinkingcap-20260924T111028Z-p8192.jsonl. Production needs 0019 before it next deploys.
With 0019, 8 of 8 suites PASS stages 0-3 (ThinkingCap 4/4, Qwen 4/4). The checks covered:
- a 20-turn agentic tool loop and the canary
- 32 requests at concurrency 16, all correct
- determinism, cold/cached prefill equality, and batch invariance at [1, 8, 32]
- haystack retrieval at 8k and 32k
- the stage-4 audit found 0 degenerate completions
GSM8K with thinking on: ThinkingCap 57/60 vs Qwen 58/60, with 0 truncated. n=60 cannot resolve a 1-item difference.

Speed (TG = ITL p50; TTFT = the last stream of the burst)

rung	B	TTFT max s (TC / Qwen)	makespan s (TC / Qwen)	TG ms/tok (TC / Qwen)	prefill tok/s
4k	4	7.48 / 7.32	60.2 / 61.4	50.7 / 51.8	2,260
4k	8	14.68 / 14.63	71.6 / 71.3	53.4 / 53.6	2,250
8k	4	11.70 / 11.75	61.8 / 66.2	47.9 / 52.3	2,820
8k	8	23.24 / 23.23	80.1 / 79.8	52.9 / 53.5	2,845
16k	4	20.77 / 20.93	76.8 / 77.1	54.0 / 54.2	3,170
16k	8	41.58 / 41.58	101.3 / 101.5	56.8 / 58.1	3,170
32k	4	39.64 / 39.69	97.6 / 98.4	56.4 / 56.7	3,250
32k	8	79.30 / 79.46	143.0 / 144.9	61.3 / 62.5	3,245

Single-stream cold TTFT is 1.84 / 2.95 / 5.28 / 10.03 s at 4k / 8k / 16k / 32k, the same for both checkpoints.
Against the owner's bands:
- TTFT under 5 s ("good") holds only for a single stream at 8k or less.
- TG runs at 16-21 tok/s per stream, against a "good" band of 70 or more.
- SLO attainment is 0.00 at every B=4/8 cell for both models, because batched prefill is serialised (it shows as a TTFT staircase).
The two checkpoints run at the same speed, within 0-2%, as expected for the same architecture.

Where ThinkingCap wins: fewer thinking tokens

GSM8K, thinking on, B=8	ThinkingCap	Qwen	delta
completion tokens, median	145	224	-35%
completion tokens, mean	185	387	-52%
time-to-answer, median	17.2 s	20.2 s	-15%
time-to-answer, mean	20.8 s	32.0 s	-35%
wall time for 60 items	201 s	344 s	-42%

For comparison only (THIRD-PARTY): the ThinkingCap model card reports a median of 484 -> 232 tokens on MMLU-Pro on H200. Our result points the same way, with a smaller effect, on a different task.

Caveats

infbench stage 4 always runs with thinking off and uses synthetic prompts. Thinking-on behaviour was measured only through GSM8K.
Output length differs between the checkpoints, which moves e2e tok/s. Rank speed on ITL and TTFT instead.
The arms ran sequentially, ThinkingCap then Qwen. Repeat spread within each arm was 1% or less.
max_model_len was 34816, so these results compare with the 2026-09-14 sweep shape, not with production records that used 262144.
bench/ab/sweep-32k.sh was not run. It uses the goal1 profile, and the infbench rungs already cover the same grid on the production arm.

This is the no-MTP baseline for #63 (MTP on ThinkingCap) and the model switch (ledger `0aaaaaaaaaaaaaaaaaa`). Related: #62. All numbers are **MEASURED** on cfx-llm2 (2x P150a, TP=2) on 2026-09-24. Branch: `thinkingcap-baseline` (855ef82). Summary record: `bench/runs/tc-baseline-summary-20260924.jsonl`. ## Setup - Both checkpoints ran on one source tree and profile. Only the checkpoint and its tensor cache changed. - Profile: `scripts/vllm-tt-serve.sh prod-argmax` (bf8 KV, device argmax, no prefix caching, `trace_mode=all`). - Server: `CTX=34816 SEQS=8`, pool 1,048,576, `vllm-tt:k2`. - Sources: the **deployed** `tenstorrent-serving` nix-store set plus the hand-ported **0019** slot fix. The tool parser is `qwen3_xml`. - ThinkingCap got its own cache (`tt-cache/thinkingcap-e55ba7f6-prod`), built with `COLD=1`. The MTP harness's cache lacked the bf4 `down_proj` tensors this profile needs, so it was not reused. - Every boot's server log was checked for these banners: `QWEN_SDPA_BF8 engaged (BFLOAT8_B)`, `QWEN36_DEVICE_ARGMAX engaged`, `QWEN36_PREFILLED_SLOTS_FIX engaged`, `GPU KV cache size: 1,049,088`. The `/v1/models` `root` matched the checkpoint in each case. - Harness: infbench stages 0-4, one gated suite per rung (`bench/infbench/tc-baseline/*.toml`). - Load: synchronised burst of B = 4 and 8, 1024 max output tokens, 2 repeats. - Quality: `bench/gsm8k.py` with thinking **on**, n=60, B=8, `max_tokens` 8192. ## Correctness - **The deployed production argmax tree fails stage 3 without 0019.** The concurrency check failed 1 of 32 requests with `expected 391, got 33`. That is the same slot-reuse signature 0019 fixed on 09-19. The deployed unit carries 0014 but not 0019. Record: `tc-baseline-thinkingcap-20260924T111028Z-p8192.jsonl`. Production needs 0019 before it next deploys. - **With 0019, 8 of 8 suites PASS stages 0-3** (ThinkingCap 4/4, Qwen 4/4). The checks covered: - a 20-turn agentic tool loop and the canary - 32 requests at concurrency 16, all correct - determinism, cold/cached prefill equality, and batch invariance at [1, 8, 32] - haystack retrieval at 8k and 32k - the stage-4 audit found 0 degenerate completions - GSM8K with thinking on: **ThinkingCap 57/60 vs Qwen 58/60**, with 0 truncated. n=60 cannot resolve a 1-item difference. ## Speed (TG = ITL p50; TTFT = the last stream of the burst) | rung | B | TTFT max s (TC / Qwen) | makespan s (TC / Qwen) | TG ms/tok (TC / Qwen) | prefill tok/s | |---|---|---|---|---|---| | 4k | 4 | 7.48 / 7.32 | 60.2 / 61.4 | 50.7 / 51.8 | 2,260 | | 4k | 8 | 14.68 / 14.63 | 71.6 / 71.3 | 53.4 / 53.6 | 2,250 | | 8k | 4 | 11.70 / 11.75 | 61.8 / 66.2 | 47.9 / 52.3 | 2,820 | | 8k | 8 | 23.24 / 23.23 | 80.1 / 79.8 | 52.9 / 53.5 | 2,845 | | 16k | 4 | 20.77 / 20.93 | 76.8 / 77.1 | 54.0 / 54.2 | 3,170 | | 16k | 8 | 41.58 / 41.58 | 101.3 / 101.5 | 56.8 / 58.1 | 3,170 | | 32k | 4 | 39.64 / 39.69 | 97.6 / 98.4 | 56.4 / 56.7 | 3,250 | | 32k | 8 | 79.30 / 79.46 | 143.0 / 144.9 | 61.3 / 62.5 | 3,245 | - Single-stream cold TTFT is **1.84 / 2.95 / 5.28 / 10.03 s** at 4k / 8k / 16k / 32k, the same for both checkpoints. - Against the owner's bands: - TTFT under 5 s ("good") holds only for a single stream at 8k or less. - TG runs at 16-21 tok/s per stream, against a "good" band of 70 or more. - SLO attainment is **0.00** at every B=4/8 cell for both models, because batched prefill is serialised (it shows as a TTFT staircase). - The two checkpoints run at the same speed, within 0-2%, as expected for the same architecture. ## Where ThinkingCap wins: fewer thinking tokens | GSM8K, thinking on, B=8 | ThinkingCap | Qwen | delta | |---|---|---|---| | completion tokens, median | 145 | 224 | -35% | | completion tokens, mean | 185 | 387 | -52% | | time-to-answer, median | 17.2 s | 20.2 s | -15% | | time-to-answer, mean | 20.8 s | 32.0 s | -35% | | wall time for 60 items | 201 s | 344 s | -42% | For comparison only (**THIRD-PARTY**): the ThinkingCap model card reports a median of 484 -> 232 tokens on MMLU-Pro on H200. Our result points the same way, with a smaller effect, on a different task. ## Caveats - infbench stage 4 always runs with thinking **off** and uses synthetic prompts. Thinking-on behaviour was measured only through GSM8K. - Output length differs between the checkpoints, which moves e2e tok/s. Rank speed on ITL and TTFT instead. - The arms ran sequentially, ThinkingCap then Qwen. Repeat spread within each arm was 1% or less. - `max_model_len` was 34816, so these results compare with the 2026-09-14 sweep shape, not with production records that used 262144. - `bench/ab/sweep-32k.sh` was not run. It uses the goal1 profile, and the infbench rungs already cover the same grid on the production arm.

Grok referenced this issue

2026-09-24 14:59:35 +02:00

MTP on ThinkingCap-Qwen3.8-27B-FP8 (TP=2): inventory, per-row B=4/8 design, go/no-go, phased gates #63

Grok referenced this issue

2026-09-24 21:54:47 +02:00

DSpark drafter (ThatchCloud) on ThinkingCap #66

Grok referenced this issue

2026-09-24 22:43:02 +02:00

DSpark drafter (ThatchCloud) on ThinkingCap #66

Grok referenced this issue

2026-09-25 02:22:40 +02:00

DSpark drafter (ThatchCloud) on ThinkingCap #66