bitpartner/tt-stack

Watch

Fork 0

prefix caching: the unaligned-resume equivalence gate dissolves — the path is unreachable #47

New issue

Open

opened 2026-09-20 08:55:10 +02:00 by Grok · 2 comments

Grok commented

2026-09-20 08:55:10 +02:00

Owner

Summary

docs/LOOP-ITERATIVE-IMPROVEMENT.md carried backlog lever 2, "unaligned-resume
equivalence — required before prefix caching ships", as outstanding correctness work
gating the prefix-caching arm. It is not work. The condition it guards cannot occur
in any valid configuration. Recorded here so it is not picked up again.

Verdict record: bench/runs/unaligned-resume-equivalence-verdict-20260920T0800Z.jsonl
(4f2cdc2). No hardware was used and no boot was spent.

The argument

Three enforced constraints compose:

qwen36_prefix_cache.py:47 — if block_size < 2048 or block_size % 2048: raise,
so block_size is a multiple of 2048.
prefill_chunk_plan validation — chunk_size % block_size must be 0, which
rejects any block_size larger than chunk_size.
model.py:2107 asserts the 2048-token chunk, and every call site passes 2048
(894, 915, 1142, 1258, 2548, 3437).

Enumerating 64…32768, constraints 1 and 2 with chunk_size = 2048 admit exactly one
legal block size: 2048.

qwen36_prefix_cache.py:28-30 — a resume start is a prefix-cache hit length and is
therefore always a multiple of block_size.

Therefore start % chunk_size == 0 always, and plan["lead"] is always None.
Exhaustive check over the legal space: 0 of 1,400 (block_size, start, end)
combinations reach the unaligned lead.

Do NOT delete the lead branch

It is defensive, not dead-by-accident. It is the only thing standing between a
future chunk_size change and a silent KV gap.

Re-open trigger, explicitly

Raising chunk_size above 2048 while block_size stays 2048 makes the path live
immediately:

prefill_chunk_plan(2048, 9000, chunk_size=4096, block_size=2048)
# -> lead=(2048, 2048)

Any change to the 2048 chunk must re-open this lever. The model.py:2107 assert is
what currently prevents it.

Evidence the instrument is not vacuous

bench/fixtures/fixtures-prefix-chunk-plan-test.py passes 12 cases against the
real prefix model.py, including three unaligned-resume shapes ([1024,8222),
[64,2048), [2112,9000)), and its negative control rejects the unpatched logic
on all 8 resume cases.
bench/prefix_hit_equivalence.py passes 7/7 fixture cases including the BLIND
detector (a harness that gives the same digest to a different question is caught).

The one weaker link, stated

Link 4 (hit lengths are whole blocks) is verified from the in-source comment at
qwen36_prefix_cache.py:28-30 plus vLLM's standard block-granular hit accounting —
not by executing the hit path. It is the only one of the four not proven by direct
enumeration.

What this does NOT cover

Aligned-resume numerical equivalence is a different question and remains open. It is
what bench/ab/cache-speed-16k-32k.sh's equiv_cold / equiv_warm / equiv_control
phases exercise, and that has not been run on the current tree. That is the real
remaining prefix correctness item.

Related: #47

## Summary `docs/LOOP-ITERATIVE-IMPROVEMENT.md` carried backlog lever 2, *"unaligned-resume equivalence — required before prefix caching ships"*, as outstanding correctness work gating the prefix-caching arm. **It is not work.** The condition it guards cannot occur in any valid configuration. Recorded here so it is not picked up again. Verdict record: `bench/runs/unaligned-resume-equivalence-verdict-20260920T0800Z.jsonl` (`4f2cdc2`). No hardware was used and no boot was spent. ## The argument Three enforced constraints compose: 1. `qwen36_prefix_cache.py:47` — `if block_size < 2048 or block_size % 2048: raise`, so `block_size` is a multiple of 2048. 2. `prefill_chunk_plan` validation — `chunk_size % block_size` must be 0, which **rejects** any `block_size` larger than `chunk_size`. 3. `model.py:2107` asserts the 2048-token chunk, and every call site passes 2048 (894, 915, 1142, 1258, 2548, 3437). Enumerating 64…32768, constraints 1 and 2 with `chunk_size = 2048` admit **exactly one** legal block size: **2048**. 4. `qwen36_prefix_cache.py:28-30` — a resume `start` is a prefix-cache hit length and is therefore always a multiple of `block_size`. Therefore `start % chunk_size == 0` always, and `plan["lead"]` is always `None`. Exhaustive check over the legal space: **0 of 1,400** `(block_size, start, end)` combinations reach the unaligned lead. ## Do NOT delete the lead branch It is **defensive, not dead-by-accident**. It is the only thing standing between a future `chunk_size` change and a silent KV gap. ## Re-open trigger, explicitly Raising `chunk_size` above 2048 while `block_size` stays 2048 makes the path live immediately: ```python prefill_chunk_plan(2048, 9000, chunk_size=4096, block_size=2048) # -> lead=(2048, 2048) ``` Any change to the 2048 chunk must re-open this lever. The `model.py:2107` assert is what currently prevents it. ## Evidence the instrument is not vacuous - `bench/fixtures/fixtures-prefix-chunk-plan-test.py` passes **12 cases** against the real prefix `model.py`, including three unaligned-resume shapes (`[1024,8222)`, `[64,2048)`, `[2112,9000)`), and its **negative control rejects the unpatched logic on all 8 resume cases**. - `bench/prefix_hit_equivalence.py` passes **7/7** fixture cases including the BLIND detector (a harness that gives the same digest to a different question is caught). ## The one weaker link, stated Link 4 (hit lengths are whole blocks) is verified from the in-source comment at `qwen36_prefix_cache.py:28-30` plus vLLM's standard block-granular hit accounting — **not** by executing the hit path. It is the only one of the four not proven by direct enumeration. ## What this does NOT cover **Aligned**-resume numerical equivalence is a different question and remains open. It is what `bench/ab/cache-speed-16k-32k.sh`'s `equiv_cold` / `equiv_warm` / `equiv_control` phases exercise, and that has **not** been run on the current tree. That is the real remaining prefix correctness item. Related: #47

Grok referenced this issue

2026-09-20 08:55:31 +02:00

prefix caching: deploy decision — the correctness gate is clear, this is the largest remaining makespan win #48

Grok commented

2026-09-20 08:55:36 +02:00

Author

Owner

Correction: the "Related: #47" line in the description is a self-reference typo. The intended link is #48 (the prefix-caching deploy decision), which is where the remaining open item from the last section — aligned-resume numerical equivalence on the current tree — is tracked.

Correction: the "Related: #47" line in the description is a self-reference typo. The intended link is **#48** (the prefix-caching deploy decision), which is where the remaining open item from the last section — aligned-resume numerical equivalence on the current tree — is tracked.

Grok referenced this issue from a commit

2026-09-24 20:03:10 +02:00

model: fail closed on unaligned prefix resume (#47)

Grok commented

2026-09-24 20:03:21 +02:00

Author

Owner

agy research (2026-09-24)

Findings: Unaligned-Resume Path is Truly Unreachable

Independently verified the prefix-caching resume path across the vLLM plugin and TT model:

qwen36_prefix_cache.py:254: Enforces block_size >= 2048 and block_size % 2048 == 0.
prefill_chunk_plan (model.py): Enforces chunk_size % block_size == 0.
model.py:2120 & qwen36_vllm.py:31: Pin chunk_size == 2048.
- Constraints 1–3 admit EXACTLY ONE legal block size: 2048.
Resume start is a prefix hit length (cp.end), which prepare() strictly requires to match a block checkpoint (cp.end == start where checkpoints only record at end % block_size == 0). Thus start % 2048 == 0 always.
Therefore start % chunk_size == 0 holds for every valid call; plan["lead"] is None across all 2,102 enumerated legal (start, end) configurations.
Re-open trigger: Raising chunk_size above 2048 (e.g. chunk_size=4096, block_size=2048, start=2048) makes the lead branch live immediately (lead=(2048, 2048)).
Recommendation: CLOSE Issue #47 (gate dissolved; lever struck from backlog as blocked work).

What Changed

Branch: agy/issue-47
Commit: 4c65566
Changes:
- Added fail-closed assertions (assert plan["lead"] is None / assert False) citing Issue #47 across prefill_traced_chunked, _prefill_chunked_eager_tp, and _prefill_traced_chunked_tp in model.py, while leaving defensive lead execution logic intact.
- Added host-only verification fixture bench/fixtures/fixtures-prefix-unaligned-resume-test.py covering exhaustive parameter sweeps (2,102 cases), constraint uniqueness, input rejection, re-open trigger, and loud crash on non-None lead.

Hardware Measurement Needed Next

Issue #47: NONE. Offline invariant proof; 0 device minutes spent.
Prefix Caching Deploy (#48): Aligned-resume numerical equivalence on the current tree:
- Command: bench/ab/cache-speed-16k-32k.sh
- Expected result: 100% token-identical greedy output between equiv_cold and equiv_warm (B=1/B=4), with warm TTFT beating nohead above the relative floor.
- Go/no-go: 1.00 token match -> GO for prefix caching (#48); any greedy token divergence -> NO-GO.

Open Questions

Are there any plans to increase _chunked_chunk_size above 2048? (Any such change will hit the fail-closed assertion and must re-open the unaligned-resume qualification lever).

**agy research (2026-09-24)** ### Findings: Unaligned-Resume Path is Truly Unreachable Independently verified the prefix-caching resume path across the vLLM plugin and TT model: 1. `qwen36_prefix_cache.py:254`: Enforces `block_size >= 2048 and block_size % 2048 == 0`. 2. `prefill_chunk_plan` ([model.py](patches/prod-local-metal/models/demos/blackhole/qwen36/tt/model.py#L64)): Enforces `chunk_size % block_size == 0`. 3. `model.py:2120` & `qwen36_vllm.py:31`: Pin `chunk_size == 2048`. - Constraints 1–3 admit EXACTLY ONE legal block size: `2048`. 4. Resume `start` is a prefix hit length (`cp.end`), which `prepare()` strictly requires to match a block checkpoint (`cp.end == start` where checkpoints only record at `end % block_size == 0`). Thus `start % 2048 == 0` always. 5. Therefore `start % chunk_size == 0` holds for every valid call; `plan["lead"]` is `None` across all 2,102 enumerated legal `(start, end)` configurations. 6. Re-open trigger: Raising `chunk_size` above 2048 (e.g. `chunk_size=4096, block_size=2048, start=2048`) makes the lead branch live immediately (`lead=(2048, 2048)`). **Recommendation**: CLOSE Issue #47 (gate dissolved; lever struck from backlog as blocked work). ### What Changed - Branch: `agy/issue-47` - Commit: `4c65566` - Changes: - Added fail-closed assertions (`assert plan["lead"] is None` / `assert False`) citing Issue #47 across `prefill_traced_chunked`, `_prefill_chunked_eager_tp`, and `_prefill_traced_chunked_tp` in [model.py](patches/prod-local-metal/models/demos/blackhole/qwen36/tt/model.py#L2568), while leaving defensive lead execution logic intact. - Added host-only verification fixture [bench/fixtures/fixtures-prefix-unaligned-resume-test.py](bench/fixtures/fixtures-prefix-unaligned-resume-test.py) covering exhaustive parameter sweeps (2,102 cases), constraint uniqueness, input rejection, re-open trigger, and loud crash on non-`None` lead. ### Hardware Measurement Needed Next - **Issue #47**: NONE. Offline invariant proof; 0 device minutes spent. - **Prefix Caching Deploy (#48)**: Aligned-resume numerical equivalence on the current tree: - Command: `bench/ab/cache-speed-16k-32k.sh` - Expected result: 100% token-identical greedy output between `equiv_cold` and `equiv_warm` (B=1/B=4), with warm TTFT beating nohead above the relative floor. - Go/no-go: 1.00 token match -> GO for prefix caching (#48); any greedy token divergence -> NO-GO. ### Open Questions - Are there any plans to increase `_chunked_chunk_size` above 2048? (Any such change will hit the fail-closed assertion and must re-open the unaligned-resume qualification lever).

No labels

human-approved

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack#47

No description provided.

Rows
Columns