DSpark drafter (ThatchCloud) on ThinkingCap #66

Open
opened 2026-09-24 21:54:47 +02:00 by Grok · 3 comments
Owner

Branch: dspark-thinkingcap. Related: #23 (the old DSpark acceptance probe), #63 (MTP on ThinkingCap), #65 (ThinkingCap baseline).
Labels: MEASURED (ours, with a record), THIRD-PARTY, ESTIMATE.

ThatchCloud's latest state (THIRD-PARTY, repo Thatch-cloud/Tenstorrent.Blackhole-Qwen3.8-27B, pushed 2026-09-24)

  • Branch tips:

    • newest: experiment/t32-score-reuse @ e98dbb0;
    • tags run up to lever-n-m3native-v224;
    • main: 83c9562;
    • ci/qwen-hardware-correctness is stale (3b62f7f, 09-12).
  • Offline combined T16/DSpark runtime, one stream, synthetic prompt: 128.65 TG / 3,320 PP at 4k. Context ladder:

    Context 4k 8k 16k 32k 64k 131k 262k
    TG 118.2 106.5 87.1 90.0 50.5 53.7 fails
  • Real text (docs/real-text-2026-09-24.md):

    • 4.4-5.3 tokens per full-draft T16 round (synthetic gave 6.8);
    • per-position P(accepted >= k) ≈ .82/.71/.56/.44/.35/.32/.23 for k = 1..7;
    • refactoring 7-9.5 per round; review/explanation 2.9-5.7;
    • single stream: 47.7 tok/s at 32k (106 ms/round), 32.5 tok/s at 131k.
  • 4-user "m3native" packed path (uses DFlash2, not DSpark): ~20-23.6 tok/s per user while packed, but only 24-39 tok/s aggregate delivered, because most steps fall back with fewer than 4 live users.

  • Costs: T16 verify ~59-63 ms, draft ~21 ms/block, commit 10-21 ms.

  • GDN is handled commit-only (per-token state history, DMA commit of the accepted slot, no rollback).

  • Their stack is their own tree: tt-metal 9f9cd4fd, target Qwen/Qwen3.8-27B, their own P150-pair mesh descriptor.

  • The drafter is the same RadixArk Qwen3.8-27B-DSpark @ b9a5dbdf we already hold.

  • No ThinkingCap-specific drafter exists anywhere; the ThinkingCap card mentions only MTP k=3.

  • RadixArk card: acceptance length 3.43 (GB300, NVFP4 target, thinking on, temp 1); H200 speedup 2.25-3.16x at concurrency 1, 1.09-1.75x at 32.

Their B=4 path loses to our plain decode. 4 users x 256 tokens at 32k: our plain B=4 decode takes ~14.8 s (57.67 ms/step, MEASURED, bench/runs/sweep-32k-20260914T142154Z.jsonl), their packed spec path 26.2 s (THIRD-PARTY). Their win is single-stream latency, not batched makespan.

Usable on ThinkingCap? Verdict: usable in principle; the value is unmeasured; port as an oracle first

What the drafter consumes

  • Target taps at layers 5/19/33/47/61 (hidden_states[l+1]), concatenated to 25600, then fc.
  • The target's embed and lm_head.
  • ThinkingCap's embed and lm_head are byte-identical to Qwen's. Its linear weights differ by 1-4%, so the taps shift slightly. Acceptance on ThinkingCap therefore has to be measured, not assumed; it is the one open quantity.

What runs at TP=2

  • The drafter is 1.86B params, bf16 ~1.85 GB/chip if replicated.
  • 5 full-attention layers with GQA 32/8 heads, which shard 16/4 per chip like the target.
  • Its KV is ~100 KB/token in bf16, ~3x the target's bf8 KV. It fits at ≤32k x B=4. It does not fit at 8x128k / 4x256k without a window or bf8 (the drafter was trained full-history, so a window costs acceptance).

What it needs from our 0.79 tree

  • Tap readout: hookable today.
  • A K+1-row verify forward at TP=2.
  • A per-row GDN commit.
  • Plugin plumbing.
  • These are the same missing pieces as MTP #63 (its P2/P3). Thatch's tree cannot be lifted: different pin, target and runtime.

Against MTP (#63): ESTIMATE, same verifier, so the comparison is draft cost against acceptance

MTP k=3 (#63 go/no-go ESTIMATE) DSpark K=7 (ESTIMATE from Thatch real-text P(acc>=k))
tokens/round per #63 go/no-go (acceptance still UNMEASURED) ~4.4 (Qwen base, their tree); ~3.1 at K=3
draft cost/round 1 MTP layer x k (~few ms) 5-layer drafter, one block (~21 ms THIRD-PARTY on their tree)
B=1 makespan -39% (x2.01) similar or better if verify(8 rows) ≈ verify(4 rows); draft cost eats part of it
B=4 -26% (x1.79-1.84) smaller: 8 verify rows x 4 users = 32 rows/step
B=8 -17%..-29% worst: 64 rows, and the drafter KV does not fit at the target shapes
effort weights in checkpoint; one layer +1.86B model port, own KV pool, tap plumbing

Order: MTP first (cheaper draft, KV-free). DSpark is worth building only if the oracle shows ThinkingCap acceptance clearly above MTP's, and only for B≤4 / ≤32k.

#23: why its 0.10 was a harness defect (confirming and extending agy's advisory)

  1. ctx_len=1 (agy).
  2. RoPE off by one (agy).
  3. New: agy's fix replays the last W prompt tokens through decode after prefill. That advances the 48 in-place GDN recurrent and conv states a second time, so the target itself is corrupted.
  4. New: the prompt is a random closed-vocabulary word salad with no chat template.
  5. New: a W-token context window on a drafter trained with full-history attention understates acceptance by construction.

Plan / done so far

  • Host teacher-forced oracle bench/dspark/oracle.py:
    • real drafter, cached per-layer context K/V, SpecForge accounting;
    • K sweep; shift±1, wN and perfect controls; think/answer split;
    • 27 host tests pass; the cache is equivalent to DSparkReference at every length and window.
  • Device capture bench/dspark/tap_capture.py + run-capture.sh:
    • production tree (vllm-tt:k2 + src19 mounts, prod-argmax env, thinkingcap-e55ba7f6-prod cache);
    • chat-templated agentic/code/free_text prompts;
    • fail-closed embedding-order and warm-cache checks;
    • phase-2 multi-row taps.
  • Capture ThinkingCap plus a Qwen control arm (queued under /tmp/ttlock, not preempting).
  • Oracle acceptance, MEASURED, agentic vs free_text reported separately, with controls.
  • Only if acceptance beats MTP: verify cost and fixed-width greedy identity, which need the #63 TP=2 verifier.
Branch: `dspark-thinkingcap`. Related: #23 (the old DSpark acceptance probe), #63 (MTP on ThinkingCap), #65 (ThinkingCap baseline). Labels: **MEASURED** (ours, with a record), **THIRD-PARTY**, **ESTIMATE**. ## ThatchCloud's latest state (THIRD-PARTY, repo `Thatch-cloud/Tenstorrent.Blackhole-Qwen3.8-27B`, pushed 2026-09-24) - Branch tips: - newest: `experiment/t32-score-reuse` @ `e98dbb0`; - tags run up to `lever-n-m3native-v224`; - main: `83c9562`; - `ci/qwen-hardware-correctness` is stale (`3b62f7f`, 09-12). - Offline combined T16/DSpark runtime, one stream, synthetic prompt: **128.65 TG / 3,320 PP at 4k**. Context ladder: | Context | 4k | 8k | 16k | 32k | 64k | 131k | 262k | |---|---|---|---|---|---|---|---| | TG | 118.2 | 106.5 | 87.1 | 90.0 | 50.5 | 53.7 | fails | - Real text (`docs/real-text-2026-09-24.md`): - **4.4-5.3 tokens per full-draft T16 round** (synthetic gave 6.8); - per-position P(accepted >= k) ≈ .82/.71/.56/.44/.35/.32/.23 for k = 1..7; - refactoring 7-9.5 per round; review/explanation 2.9-5.7; - single stream: 47.7 tok/s at 32k (106 ms/round), 32.5 tok/s at 131k. - 4-user "m3native" packed path (uses **DFlash2**, not DSpark): ~20-23.6 tok/s per user while packed, but only **24-39 tok/s aggregate delivered**, because most steps fall back with fewer than 4 live users. - Costs: T16 verify ~59-63 ms, draft ~21 ms/block, commit 10-21 ms. - GDN is handled **commit-only** (per-token state history, DMA commit of the accepted slot, no rollback). - Their stack is their own tree: tt-metal `9f9cd4fd`, target Qwen/Qwen3.8-27B, their own P150-pair mesh descriptor. - The drafter is the same RadixArk `Qwen3.8-27B-DSpark` @ `b9a5dbdf` we already hold. - **No ThinkingCap-specific drafter exists** anywhere; the ThinkingCap card mentions only MTP k=3. - RadixArk card: acceptance length 3.43 (GB300, NVFP4 target, thinking on, temp 1); H200 speedup 2.25-3.16x at concurrency 1, 1.09-1.75x at 32. **Their B=4 path loses to our plain decode.** 4 users x 256 tokens at 32k: our plain B=4 decode takes ~14.8 s (57.67 ms/step, MEASURED, `bench/runs/sweep-32k-20260914T142154Z.jsonl`), their packed spec path 26.2 s (THIRD-PARTY). Their win is single-stream latency, not batched makespan. ## Usable on ThinkingCap? Verdict: usable in principle; the value is unmeasured; port as an oracle first **What the drafter consumes** - Target taps at layers 5/19/33/47/61 (hidden_states[l+1]), concatenated to 25600, then `fc`. - The target's embed and lm_head. - **ThinkingCap's embed and lm_head are byte-identical to Qwen's.** Its linear weights differ by 1-4%, so the taps shift slightly. Acceptance on ThinkingCap therefore has to be measured, not assumed; it is the one open quantity. **What runs at TP=2** - The drafter is 1.86B params, bf16 ~1.85 GB/chip if replicated. - 5 full-attention layers with GQA 32/8 heads, which shard 16/4 per chip like the target. - Its KV is **~100 KB/token in bf16, ~3x the target's bf8 KV**. It fits at ≤32k x B=4. It does **not** fit at 8x128k / 4x256k without a window or bf8 (the drafter was trained full-history, so a window costs acceptance). **What it needs from our 0.79 tree** - Tap readout: hookable today. - A K+1-row verify forward at TP=2. - A per-row GDN commit. - Plugin plumbing. - **These are the same missing pieces as MTP #63 (its P2/P3).** Thatch's tree cannot be lifted: different pin, target and runtime. **Against MTP (#63): ESTIMATE, same verifier, so the comparison is draft cost against acceptance** | | MTP k=3 (#63 go/no-go ESTIMATE) | DSpark K=7 (ESTIMATE from Thatch real-text P(acc>=k)) | |---|---|---| | tokens/round | per #63 go/no-go (acceptance still UNMEASURED) | ~4.4 (Qwen base, their tree); ~3.1 at K=3 | | draft cost/round | 1 MTP layer x k (~few ms) | 5-layer drafter, one block (~21 ms THIRD-PARTY on their tree) | | B=1 makespan | -39% (x2.01) | similar or better if verify(8 rows) ≈ verify(4 rows); draft cost eats part of it | | B=4 | -26% (x1.79-1.84) | smaller: 8 verify rows x 4 users = 32 rows/step | | B=8 | -17%..-29% | worst: 64 rows, and the drafter KV does not fit at the target shapes | | effort | weights in checkpoint; one layer | +1.86B model port, own KV pool, tap plumbing | Order: **MTP first** (cheaper draft, KV-free). DSpark is worth building only if the oracle shows ThinkingCap acceptance clearly above MTP's, and only for B≤4 / ≤32k. ## #23: why its 0.10 was a harness defect (confirming and extending agy's advisory) 1. ctx_len=1 (agy). 2. RoPE off by one (agy). 3. **New:** agy's fix replays the last W prompt tokens through decode *after* prefill. That advances the 48 in-place GDN recurrent and conv states a second time, so the target itself is corrupted. 4. **New:** the prompt is a random closed-vocabulary word salad with no chat template. 5. **New:** a W-token context window on a drafter trained with full-history attention understates acceptance by construction. ## Plan / done so far - [x] Host teacher-forced oracle `bench/dspark/oracle.py`: - real drafter, cached per-layer context K/V, SpecForge accounting; - K sweep; `shift±1`, `wN` and `perfect` controls; think/answer split; - 27 host tests pass; the cache is equivalent to `DSparkReference` at every length and window. - [x] Device capture `bench/dspark/tap_capture.py` + `run-capture.sh`: - production tree (`vllm-tt:k2` + src19 mounts, prod-argmax env, `thinkingcap-e55ba7f6-prod` cache); - chat-templated agentic/code/free_text prompts; - fail-closed embedding-order and warm-cache checks; - phase-2 multi-row taps. - [ ] Capture ThinkingCap plus a Qwen control arm (queued under `/tmp/ttlock`, not preempting). - [ ] Oracle acceptance, MEASURED, agentic vs free_text reported separately, with controls. - [ ] Only if acceptance beats MTP: verify cost and fixed-width greedy identity, which need the #63 TP=2 verifier.
Author
Owner

Progress: device captures done (MEASURED, bench/runs/dspark-tc-capture-20260924T195053Z.jsonl)

Setup

  • Production tree: vllm-tt:k2 + src19 mounts, prod-argmax env with host argmax, bf8 paged KV (banner), 1x2 TP=2.
  • Two arms:
    • ThinkingCap on thinkingcap-e55ba7f6-prod;
    • Qwen base on the shared cache.
  • Both arms booted from a warm build of their own cache.
  • 6 chat-templated prompts per arm: agentic x3, code, free_text x2.

Checks (all passed)

  • Embedding mesh-concat order: PCC 1.0, max diff 0.
  • The phase-2 multi-row next token matches the decoded token: 12/12 prompts.
  • Prefill is deterministic: 12/12 prompts.

Speeds. These are NOT production numbers; production is in #65.

  • Eager prefill with no trace parked: 2,138 tok/s at 2,158 prompt tokens and 2,714 tok/s at 3,394 (warm programs). Short prompts are compile-dominated.
  • Untraced eager decode: 213.5-219.4 ms/token.

Features

  • Decode-row taps vs the multi-row (verify-like) taps, float64 PCC: 0.9999+ at layers 5/19. At layers 47/61 the range is 0.996-0.9994, except the full-file re-emission prompt at 0.982-0.983.
  • ThinkingCap vs Qwen taps on identical prompts: PCC 0.9999+ at layer 5, falling to 0.987-0.998 at layer 61. The fine-tune does move the features the drafter consumes, mostly in the deep taps.
  • Greedy continuations diverge from Qwen within 0-81 tokens on 5 of 6 prompts. The no-think file re-emission is identical for all 512 tokens.

The host oracle is running on both arms now. Early ThinkingCap figure, agentic no-think file re-emission: K=7 gives 7.55 tokens/round, K=3 gives 3.94 (the perfect-drafter ceilings are 8 and 4). This is the best case, as expected. The other regimes follow.

**Progress: device captures done (MEASURED, `bench/runs/dspark-tc-capture-20260924T195053Z.jsonl`)** **Setup** - Production tree: `vllm-tt:k2` + src19 mounts, prod-argmax env with host argmax, bf8 paged KV (banner), 1x2 TP=2. - Two arms: - ThinkingCap on `thinkingcap-e55ba7f6-prod`; - Qwen base on the shared cache. - Both arms booted from a warm build of their own cache. - 6 chat-templated prompts per arm: agentic x3, code, free_text x2. **Checks (all passed)** - Embedding mesh-concat order: PCC 1.0, max diff 0. - The phase-2 multi-row next token matches the decoded token: 12/12 prompts. - Prefill is deterministic: 12/12 prompts. **Speeds. These are NOT production numbers**; production is in #65. - Eager prefill with no trace parked: 2,138 tok/s at 2,158 prompt tokens and 2,714 tok/s at 3,394 (warm programs). Short prompts are compile-dominated. - Untraced eager decode: 213.5-219.4 ms/token. **Features** - Decode-row taps vs the multi-row (verify-like) taps, float64 PCC: 0.9999+ at layers 5/19. At layers 47/61 the range is 0.996-0.9994, except the full-file re-emission prompt at 0.982-0.983. - ThinkingCap vs Qwen taps on identical prompts: PCC 0.9999+ at layer 5, falling to **0.987-0.998 at layer 61**. The fine-tune does move the features the drafter consumes, mostly in the deep taps. - Greedy continuations diverge from Qwen within 0-81 tokens on 5 of 6 prompts. The no-think file re-emission is identical for all 512 tokens. The host oracle is running on both arms now. Early ThinkingCap figure, agentic no-think file re-emission: **K=7 gives 7.55 tokens/round, K=3 gives 3.94** (the perfect-drafter ceilings are 8 and 4). This is the best case, as expected. The other regimes follow.
Author
Owner

Acceptance measured with the host oracle. Greedy teacher-forced replay of the device captures, full-history context, real RadixArk drafter (sha 2aff025f) in fp32.

Records:

  • bench/runs/dspark-tc-accept-thinkingcap-20260925.jsonl
  • bench/runs/dspark-tc-accept-qwen-20260925.jsonl

Tokens per verify round (1 + mean accepted), MEASURED, pooled over full rounds

K (verify rows = K+1) ThinkingCap agentic ThinkingCap code ThinkingCap free_text ThinkingCap ALL Qwen ALL (control)
3 3.47 2.44 2.93 3.00 2.91
7 5.21 2.94 3.66 3.95 3.79
15 6.44 3.06 4.02 4.43 4.32

Per prompt, ThinkingCap at K=7

Prompt Tokens/round
no-think full-file re-emission 7.55 (ceiling 8.0)
tool call 5.29
edit with thinking 3.98
math reasoning 4.81
code from spec 2.94
essay 2.95

Every thinking prompt spent its whole 512-token budget in <think>, so the code and free_text rows measure thinking text.

Controls at K=7 (ThinkingCap)

  • Perfect drafter: 8.0.
  • Features misaligned by ±1 position: acceptance drops on every prompt (for example 7.55 → 5.82 / 4.21).
  • Context cut to 1 or 64 positions: much lower (7.55 → 2.39 / 3.43; essay 2.95 → 2.12 / 2.49).

So the drafter really uses the full-history features, and #23-style windowing would have understated it heavily.

What this means

  • ThinkingCap does not hurt DSpark. It matches or slightly beats Qwen base (3.95 vs 3.79 at K=7) despite the deep-tap drift, so no ThinkingCap-specific drafter is needed.
  • The numbers are in line with Thatch's real-text 4.4-5.3 at T16 (THIRD-PARTY). Ours at K=15 is 4.43.
  • Quoting caveats:
    • n=1 capture per prompt, 512-token budget;
    • fp32 CPU drafter, not a TT bf16 drafter;
    • no verify, draft or commit cost measured, and no live token identity. Those need the TP=2 K+1-row verifier shared with #63.

Makespan, ESTIMATE only

  • Assumes decode ms/token t (48-62 ms, #65), verify(K+1 rows) ≈ t·(1+ε), and the ~21 ms draft block from Thatch (THIRD-PARTY).
  • B=1 at K=7: ~3.95 tokens per (t + 21 ms) ≈ 2.9x at t = 50 ms if verifying 8 rows costs about the same as 1. It falls to ~1.9x if verify costs 1.5t.
  • B=4/8: 32-64 verify rows per step, the weight-read-bound share shrinks, and the drafter KV (~100 KB/token bf16) does not fit at 8x128k / 4x256k. Expect far less, and on Thatch's evidence possibly nothing at B=4.

Next

  • Compare with MTP acceptance once #63 measures it. The expensive part, the verifier plus per-row GDN commit, is shared, so the per-round draft cost against these acceptance numbers decides it.
  • A DSpark-specific next step is the drafter forward on TT (5 layers, TP=2) to measure its real draft ms. I am not starting that before MTP's acceptance exists.
**Acceptance measured with the host oracle.** Greedy teacher-forced replay of the device captures, full-history context, real RadixArk drafter (sha `2aff025f`) in fp32. Records: - `bench/runs/dspark-tc-accept-thinkingcap-20260925.jsonl` - `bench/runs/dspark-tc-accept-qwen-20260925.jsonl` **Tokens per verify round (1 + mean accepted), MEASURED, pooled over full rounds** | K (verify rows = K+1) | ThinkingCap agentic | ThinkingCap code | ThinkingCap free_text | ThinkingCap ALL | Qwen ALL (control) | |---|---|---|---|---|---| | 3 | 3.47 | 2.44 | 2.93 | **3.00** | 2.91 | | 7 | 5.21 | 2.94 | 3.66 | **3.95** | 3.79 | | 15 | 6.44 | 3.06 | 4.02 | **4.43** | 4.32 | **Per prompt, ThinkingCap at K=7** | Prompt | Tokens/round | |---|---| | no-think full-file re-emission | 7.55 (ceiling 8.0) | | tool call | 5.29 | | edit with thinking | 3.98 | | math reasoning | 4.81 | | code from spec | 2.94 | | essay | 2.95 | Every thinking prompt spent its whole 512-token budget in `<think>`, so the code and free_text rows measure *thinking* text. **Controls at K=7 (ThinkingCap)** - Perfect drafter: 8.0. - Features misaligned by ±1 position: acceptance drops on every prompt (for example 7.55 → 5.82 / 4.21). - Context cut to 1 or 64 positions: much lower (7.55 → 2.39 / 3.43; essay 2.95 → 2.12 / 2.49). So the drafter really uses the full-history features, and #23-style windowing would have understated it heavily. **What this means** - **ThinkingCap does not hurt DSpark.** It matches or slightly beats Qwen base (3.95 vs 3.79 at K=7) despite the deep-tap drift, so no ThinkingCap-specific drafter is needed. - The numbers are in line with Thatch's real-text 4.4-5.3 at T16 (THIRD-PARTY). Ours at K=15 is 4.43. - Quoting caveats: - n=1 capture per prompt, 512-token budget; - fp32 CPU drafter, not a TT bf16 drafter; - no verify, draft or commit cost measured, and no live token identity. Those need the TP=2 K+1-row verifier shared with #63. **Makespan, ESTIMATE only** - Assumes decode ms/token t (48-62 ms, #65), verify(K+1 rows) ≈ t·(1+ε), and the ~21 ms draft block from Thatch (THIRD-PARTY). - B=1 at K=7: ~3.95 tokens per (t + 21 ms) ≈ **2.9x at t = 50 ms** if verifying 8 rows costs about the same as 1. It falls to ~1.9x if verify costs 1.5t. - B=4/8: 32-64 verify rows per step, the weight-read-bound share shrinks, and the drafter KV (~100 KB/token bf16) does not fit at 8x128k / 4x256k. Expect far less, and on Thatch's evidence possibly nothing at B=4. **Next** - Compare with MTP acceptance once #63 measures it. The expensive part, the verifier plus per-row GDN commit, is shared, so the per-round draft cost against these acceptance numbers decides it. - A DSpark-specific next step is the drafter forward on TT (5 layers, TP=2) to measure its real draft ms. I am not starting that before MTP's acceptance exists.
Author
Owner

The DSpark device port is reopened here. PR #23's "port NOT justified" is closed as superseded, because this issue's 3.95 tokens/round at K=7 beats #23's own >=3.0 threshold.

Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).

The **DSpark device port** is reopened here. PR #23's "port NOT justified" is closed as superseded, because this issue's 3.95 tokens/round at K=7 beats #23's own >=3.0 threshold. Reopened under the new rule (owner, 2026-09-25): **a closure needs an on-hardware record at the stated scope**; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).
Sign in to join this conversation.
No labels
human-approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#66
No description provided.