prefix caching: deploy decision — the correctness gate is clear, this is the largest remaining makespan win #48
Labels
No labels
human-approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack#48
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Decision needed
Prefix caching is measured, gated, and not deployed. The remaining blocker is an
owner decision, not engineering work.
Why it is the largest item on the board
Prefill is 87.6% of makespan at conc-8 on the production arm. Both cleared
optimizations (bf8 KV, device argmax) are decode-side, so neither touches it. Measured
at conc-8, 93,739-token prompts:
That is −86.6% makespan and −95.0% TTFT (
23b500a, superseded by384e9a0with device argmax + the 0019 slot fix).
This matters directly for the stated goal (4 agents @ 256k / 8 agents @ 128k). At the
target shape, prefills are fully serialised and the last agent waits 443.2 s at
8×128k and 609.2 s at 4×256k. Concurrent agents share system prompts and tool
preambles, so the shared head is exactly what those minutes are spent recomputing B
times. Prefix caching is not an optimisation here — it is the enabling feature.
Why the correctness gate is now clear
The backlog carried "unaligned-resume equivalence — required before prefix caching
ships" as the blocker. That gate dissolved: the path is unreachable in any valid
configuration (#47). No hardware was needed to establish it.
What is genuinely still open before shipping
bench/ab/cache-speed-16k-32k.shemits theequiv_cold/equiv_warm/equiv_controlphases andbench/prefix_hit_equivalence.pyis proven non-blind(7/7 fixtures incl. the BLIND detector), so the instrument exists and is trusted —
it simply has not been pointed at the tree we would ship. This is the one test I
would want green before deploying.
--no-enable-prefix-cachingis currently forced becauseqwen36_vllm.py:57declaresmodel_capabilities["supports_prefix_caching"]=False.Any doc presenting that flag as a tuning recommendation is wrong and should frame it
as a limitation.
bench/profile/prod-prefix-sources.sha256is stale — three files diverge becausevllm-tt-plugin 0004 landed after it was pinned. Do not re-pin it from patch 0018.
scripts/materialize-prod-prefix.shcurrently fails on a missingpatches/vllm/local-source/.../block_pool.py.The ask
Three options, in my order of preference:
I recommend — it converts the largest measured win in the project into production
with the one open correctness question answered.
resume is untested on this exact tree.
on the backlog exceeds ~3% of makespan, so this effectively parks the goal.
Related: #47
Addressed in PR #44 (commit
369b487). Tagged @hermes for review.Performance advisory (automated review, 2026-09-24)
I agree with option 1 (run aligned-resume equivalence, then deploy). The evidence supports two refinements, one on scoping the claim and one on which configuration to ship.
1. Quote the win as workload-dependent. −86.6% makespan / −95% TTFT (
bench/runs/prefix-traced-vs-production-pool786k-20260918T2300Z.jsonl) is a warm, 98.3%-cached, synchronised-burst measurement. The record's own caveats also include the costs:prefill-cold-pyspy-prefix-traced-20260917T193539Z.jsonlvs…-production-20260917T182851Z.jsonl). The infbench caveat gives −9.8% at an 87k head and −32% at small shapes.So the net effect depends on how much of the prompt the agents actually reuse. The owner's own test for this is
bench/ab/cache-speed-16k-32k.sh(ledger0aaaaaaaaaaaaaaaaa). Itsnoheadphase separates a real cache hit from intra-batch sharing, and itsequiv_*phases are the aligned-resume check. One boot of the positive arm answers both open items. Real agents get most of their reuse from per-agent multi-turn history (turn N re-sends turn N−1), not only from the shared system prompt. A 2–3-turn replay per agent is the realistic shape to add.2. The next bottleneck after deploying is the warm-hit tail, and it is cheap to attack. In
bench/runs/phase2a-prefix-e2e.jsonla hit computes 1,579 tokens but takes 1.94 s (814 tok/s), 3–4× slower per token than cold chunks. At conc-8 those tails serialise: TTFT max 13.9 s ≈ 8 × 1.74 s, and that is 37% of the 37.7 s makespan. From source:prefill_traced_chunked→num_full == 0→ eagerprefill_masked_bucket, the ~0.42 s host-dispatch floor. That is #50.to_torch/from_torch(tilize/untilize on host) in_snapshot_gdn_scratch/_restore_gdn_scratch(model.py:1477-1510). The cold prefix py-spy already shows 16.2% host tensor marshalling.ttnntensors in TILE layout (ttnn.from_device, thencopy_host_to_device_tensor), so a restore is a DMA and not a Python tilize. Validate first with one py-spy of a single warm request, reusing theprefill-cold-pyspyharness.3. Ship the configuration that has the correctness bound.
prefix-argmax-correctness-fail-20260919T0100Z.jsonl).prefix-intermittent-bound-20260919T0230Z.jsonl).deviceResidentassertsdeviceArgmax(modules/tenstorrent-serving.nix:544), so a resident+prefix deploy is the argmax+prefix configuration. A single 32-request pass (phase2a-prefix-e2e) cannot clear a ~1/128 rate: P(pass) ≈ 0.78 even if the defect is present.Pool check: the current prefix arm reports
kv_cache_size_tokens1,017,275, 97% of the 1,048,576 target shape. That is fine with shared heads, but 8 fully-distinct 128k agents would not quite fit.