docs: does speculative decode have a prefill analogue? #3

Merged

multica-agent merged 1 commit from prefill-optimization-notes into master

2026-09-04 12:33:38 +02:00

multica-agent commented

2026-08-20 23:13:22 +02:00

Member

Records the answer to a bring-up question: speculative decoding is our top decode lever (DECODE-OPTIMIZATION.md) — is there an equivalent for prefill?

No direct transfer, and the reason is the useful part. Spec decode reclaims idle compute created by decode's serial, bandwidth-bound dependency. Prefill has neither property — it is already parallel and compute-bound — so "draft-then-verify" has nothing to bite on. Prefill accelerators do less work rather than overlap serial work.

New docs/PREFILL-OPTIMIZATION.md records the applicable levers, ranked for our batch-1 long-context regime and graded per PERFORMANCE.md:

Prefix caching (APC) — biggest win, lossless, serving-path only.
Our GDN linear layers — a free structural prefill win (O(n), only ~16/64 layers pay the quadratic term).
Speculative prefill / token pruning (SpecPrefill, LazyLLM) — the genuine analogue, but lossy and untried on our stack; directional only.
Chunked prefill and parallel context encoding (RAG).

Plus the prefill-vs-decode "opposite optimizations" table so the two toolkits don't get mixed.

No stack measurement — this is a survey of technique, not a gate; third-party items stay graded directional. Cross-linked from DECODE-OPTIMIZATION.md.

🤖 Generated with Claude Code

Records the answer to a bring-up question: speculative decoding is our top decode lever (`DECODE-OPTIMIZATION.md`) — is there an equivalent for prefill? **No direct transfer, and the reason is the useful part.** Spec decode reclaims idle compute created by decode's *serial, bandwidth-bound* dependency. Prefill has neither property — it is already parallel and compute-bound — so "draft-then-verify" has nothing to bite on. Prefill accelerators do *less work* rather than *overlap serial work*. New `docs/PREFILL-OPTIMIZATION.md` records the applicable levers, ranked for our batch-1 long-context regime and graded per `PERFORMANCE.md`: - **Prefix caching (APC)** — biggest win, lossless, serving-path only. - **Our GDN linear layers** — a free structural prefill win (O(n), only ~16/64 layers pay the quadratic term). - **Speculative prefill / token pruning** (SpecPrefill, LazyLLM) — the genuine analogue, but lossy and untried on our stack; directional only. - **Chunked prefill** and **parallel context encoding** (RAG). Plus the prefill-vs-decode "opposite optimizations" table so the two toolkits don't get mixed. No stack measurement — this is a survey of technique, not a gate; third-party items stay graded directional. Cross-linked from `DECODE-OPTIMIZATION.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

multica-agent added 1 commit

2026-08-20 23:13:22 +02:00

docs: record whether speculative decode has a prefill analogue

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 16s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 5s

Details

4125fedfe7

We treat speculative decoding as our top decode lever; the natural next
question is whether prefill has an equivalent. It does not transfer: spec
decode reclaims idle compute created by decode's serial, bandwidth-bound
dependency, and prefill has neither — it is already parallel and
compute-bound. So prefill accelerators do less work rather than overlap
serial work, a different toolbox.

Records the applicable levers graded per PERFORMANCE.md: prefix caching
(lossless, serving-path), our GDN linear layers as a free structural win,
speculative prefill / token pruning (the genuine but lossy analogue, untried
here), chunked prefill, and parallel context encoding for RAG. No stack
measurement — survey of technique, not a gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Grok force-pushed prefill-optimization-notes from 4125fedfe7

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 16s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 5s

Details

to 0ba8bd1682

tt-stack-ci / Build simulators and check the host module (pull_request) Failing after 2s

Details

tt-stack-ci / Report upstream drift (pull_request) Successful in 5s

Details

2026-09-04 12:33:10 +02:00

Compare