research: exl3xpu / EXL3 trellis quant on Blackhole — port is no-go; EXL3-style offline bf4 quantizer + accuracy gate instead #61

Open
opened 2026-09-24 10:46:54 +02:00 by Grok · 1 comment
Owner

Desk research only. No tt-metal/vLLM process was started and the cards were not touched.
Labels used throughout: MEASURED (cites a bench/runs/* record in this repo), THIRD-PARTY
(someone else's number, not reproduced here), ESTIMATE (arithmetic done for this issue; not a result).
Standing rule: arithmetic on a measured slope is not a result. The one in-model A/B of a predicted
saving delivered 0 to -2%, and the down_proj bf4 A/B delivered 3.0% against a predicted 15.2%.

TL;DR verdict

Porting exl3xpu as it stands is a NO-GO. It would make decode slower, not faster. There are two
cheaper things worth doing instead, and both use EXL3's ideas while leaving its bitstream behind.

  1. exl3xpu is not portable code. It is an Intel Arc B70 (Xe2, "XPU" = PyTorch's Intel GPU device)
    vLLM plugin. Its kernels are SYCL/ESIMD built around Xe2 dpas (VNNI) and dp4a. It borrows GDN
    and attention from vLLM-XPU and has no tensor parallelism. Nothing in it runs on Tensix. What
    carries over is the format spec, the bit-exact PyTorch reference decoder (exl3xpu/ref.py, MIT),
    and the accuracy-gate method.
  2. Decoding the trellis on the card costs more than it saves (ESTIMATE, one cheap probe away from
    MEASURED).
    Per card, each decode step would have to turn about 12.2 B trellis weights into tiles.
    On Blackhole that work lands on the SFPU (32 lanes per Tensix; SFPMUL24 is the only integer
    multiply, and it is 23-bit). Every weight needs a 16-bit window extract, a 16x32-bit multiply
    mod 2^32, a byte-sum and one fp FMA, which is roughly 15-20 SFPU ops per 32-weight row. That puts
    the decode at ~25-50 ms per step. The most 4-bpw EXL3 could save against production's weight
    read is ~4-6 ms per step, and only if the decode were free. Break-even needs <=2 SFPU cycles
    per 32-weight row
    if decode runs serially, or <=5.6 if it overlaps the DRAM read perfectly. Doing
    the dequant in the RISC-V reader kernels is about 50x worse (~0.7 s per token).
  3. Even a free 4-bpw EXL3 barely moves the owner's metric. Weight read is 37% of the decode step.
    The bytes EXL3 would save relative to production (which already runs all MLP at bf4) are worth
    -4 to -6 ms per step. That is -4% to -10% makespan on the <=32k sweep and **
    -1% at
    8x128k / 4x256k**, where serialised prefill (443 s / 609 s) dominates. For comparison, prefix
    caching and prefill work each move those shapes by tens of percent.
  4. What is worth doing, ranked:
    • (E0) An accuracy gate for the bf4 weights we already ship. Production runs mlp.down_proj
      at bf4 (QWEN36_MLP_DOWN_BF4=1 in the prod profile). Its only record says "accuracy:
      unverified against a bf8 baseline"
      (bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl). This
      costs nothing at runtime and is needed before any weight-format change.
    • (E1) An "EXL3-inspired" offline quantizer that writes native bfp4_b. It rounds with
      Hessian awareness (GPTQ/LDLQ, EXL3's error-feedback step) straight onto the bfp4_b grid and
      emits a grid-snapped bf16 checkpoint. The host packer reproduces those bits exactly, so there
      is zero runtime change. Value: better bf4 quality where we already use it, and possibly
      enough quality to move attention/GDN projections from bf8 to bf4. That second step is worth
      about -2.6 to -4.0 ms per step and +3.4 GiB of card memory (~110k bf8 KV tokens) (ESTIMATE).
    • (E3, only if E1 falls short on some tensors) Per-tile mixed bfp8/bfp4/bfp2/zero. This uses
      Tenstorrent's own MatmulCustomCompressed plus BitSculpt "BSPM" precision maps. It is the
      native equivalent of EXL3's per-tensor bitrate recipes. It exists in tt-metal (deepseek_v3_b1)
      but has open correctness bugs at 32 cores.
  5. Running turboderp's EXL3 checkpoints on our cards is possible (option A: load-time transcode),
    but it is a compatibility feature, not a performance one.
    Transcoding EXL3 to bf8 keeps EXL3's
    quality but reads more bytes than production. Transcoding to bf4 quantizes twice and loses to our
    current source on attention/GDN.

Correction to the brief: production is not "gate/up bf4, down bf8". The prod profile sets
QWEN36_MLP_DOWN_BF4=1 (scripts/vllm-tt-serve.sh:155), so production streams ~9.33 GB/card/token.
The 10.76 GB/card figure belongs to the goal1 profile, which is what the benchmark drivers use
(bench/ab/sweep-32k.sh, decode-serving-ab.sh). Consequence: the 2026-09-14 "clean-slate"
sweep's optimized arm is not production's weight mix.
Its record's env_base has no
QWEN36_MLP_DOWN_BF4.


What exl3xpu is

  • Repo: github.com/0xSero/exl3xpu @ 2d17c57 (2026-09-24), MIT. It is a vLLM 0.26.1 XPU
    plugin (vllm.general_plugins entry point, register_quantization_config("exl3")).

  • Hardware: Intel Arc Pro B70 (BMG-G31, Xe2, 32 GB). "xpu" is PyTorch/oneAPI's Intel GPU
    device, not a generic accelerator. Kernels are SYCL ESIMD (csrc/exl3_esimd.h, 767 lines),
    JIT-compiled with icpx (spir64).

  • It consumes turboderp's EXL3 checkpoints (exllamav3 v1.x format). The served model is
    exactly ours: turboderp/Qwen3.8-27B-exl3 @ 4.00 bpw (lm_head 6 bpw, mul1 codebook, 401
    EXL3 tensors, 14.91 GiB). in_proj_a/b, norms, embeddings and the vision tower stay bf16.

  • Scope: vLLM supplies scheduling, paged KV, and the GDN and attention kernels (the XPU ones).
    exl3xpu only replaces the linear layers. No tensor parallelism ("use data parallel").
    Only 4 and 6 bpw mul1 are enabled by default; 2/3/5 bpw and the mcg/3INST codebooks need
    -DEXL3_ALL_CODEBOOKS.

  • Correctness: dequantized weights are bit-identical to exllamav3 reconstruct() for all 401
    tensors on every kernel path. Logits vs exllamav3 on an RTX 3090: top-1 99.63%, KL 9.8e-5
    on a sealed 64x256 teacher-forced panel ("Gate A3"). This is implementation equivalence
    against EXL3 itself, not quantization quality against bf16.

  • Performance, THIRD-PARTY (one B70, bench/results/2026-09-23.jsonl, recipe.json):

    exl3xpu, 1x B70 ours, 2x P150a (MEASURED)
    prefill 4k / 32k / 128k / 254k 1589 / 1497 / 1049 / 763 tok/s 3,534 tok/s @16k, 2,366 @128k, 1,707 @256k
    decode C=8 aggregate, no MTP 152 tok/s (docs/GOAL.md T4 floor) ~142.6 tok/s decode-window (B=8/4k, 56.10 ms, sweep optimized arm)
    decode C=8 aggregate, MTP k=3 (thinking on) 294-363 tok/s n/a (MTP dropped by owner)
    linears per step at M=1 / M=4 27.3 / 31.7 ms (after K=6 planar fix) weight read 23.34 ms of step (N1)

    The harnesses and definitions differ, so this is not an A/B. Our records:
    bench/runs/serving-16k-20260913T095210Z.jsonl,
    bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl,
    bench/runs/target-shape-ttft-B4-256k-20260914T134045Z.jsonl,
    bench/runs/sweep-32k-20260914T142154Z.jsonl.

  • Its own diagnosis is relevant here: on Xe2 the M=1 GEMV is co-limited by integer-ALU issue
    (~22-25 ms) and DRAM (~23.5 ms)
    . The trellis decode sits "near the integer-ALU roofline", on
    a GPU with far more integer throughput per weight than a Tensix SFPU (see below).

EXL3 format & kernel anatomy

EXL3 is turboderp's implementation of QTIP (Tseng et al., NeurIPS 2024: Quantization with
Trellises and Incoherence Processing
). The quantizer has three components, and only one of them
is tied to the bitstream:

  1. Incoherence processing. W = diag(suh) · H · W_inner · H · diag(svh), where H is the
    normalized 128x128 Sylvester Hadamard applied blockwise and suh/svh are fp16 sign/scale
    vectors. At inference: y = had128(had128(x * suh) @ W_inner) * svh. This means an input
    Hadamard and an output Hadamard per linear.
  2. Hessian-aware rounding (LDLQ). Calibration activations give H = X^T X, which is
    regularized and LDL-decomposed. Tiles are quantized in order with error feedback
    (exllamav3/modules/quant/exl3_lib/quantize.py:549 ldlq, :1471 quantize_exl3). The
    converter's proxy error is tr(E H E^T) / tr(W H W^T).
  3. Trellis-coded quantization with a computed codebook. Each 16x16 tile is 256*K bits (K =
    bpw, 1..8). Value t is the 16-bit window ending at bit (t+1)*K, circular within the
    tile, placed at a tensor-core permutation. The mul1 codebook, bit-exact per ref.py:
    x = state * 0x83DCD12D (mod 2^32), then v = fp16(1024 + bytesum(x)) * fp16(0x1EEE) + fp16(0xC931), one fp16 FMA with a single rounding.

Per weight, the inner loop is: funnel-shift two 32-bit words, mask to 16 bits, 32-bit integer
multiply, sum 4 bytes, int to fp16, one FMA. The Xe2 kernel does this in about 9 SIMD32
instructions per 32 values
: shl/shr/or/mul/dp4a/mov/mad. dp4a(0x6400, x, 0x01010101) does
the byte-sum and fp16 bias in one instruction, and that trick is the reason it is fast. Memory
access is a pure contiguous stream of trellis words per tile row (K=4 packs 8 values per word).
The 128-point Hadamards are separate small kernels (~1.2 ms per step under graphs). For M>128
(prefill) the plugin reconstructs fp16 W_inner slices and calls oneDNN GEMM.

Available bitrates: 1-8 bpw per tensor, with per-tensor recipes from sc_optimize.py. THIRD-PARTY
quality, exllamav3 doc/llama31_8b_instruct_kld_bpw.png (Llama-3.1-8B, KL vs bf16): EXL3 3.0
bpw 0.053, 3.5 bpw 0.030, 4.0 bpw 0.013, 5.0 bpw 0.003
; GGUF Q4_K_M 0.018, Q6_K 0.003. Nobody has
measured KL for RTN bfp4_b on our model. That is the gap E0/E1 close.

Qwen3.x hybrid/GDN support: yes. exllamav3 quantizes the GDN projections (in_proj_qkvz,
out_proj) like any linear, and exl3xpu serves the 48-GDN/16-attention model through vLLM's GDN
kernels. Nothing in the format is GDN-specific.

Blackhole mapping

Hardware facts (MEASURED or from source, docs/SATURATION-AND-SRAM.md): p150a is 2x-harvested,
110 compute Tensix (a column is reserved for dispatch), 1.5 MiB L1 each (165 MiB per card),
AICLK 1350 MHz, 8 GDDR6 channels. Usable DRAM is 30.87 GiB per card (docs/VRAM-BUDGET.md).
Decode matmul readers reach 454-469 GB/s (bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl).

What the Tensix pipeline gives us for free: the unpacker decompresses bfp8_b / bfp4_b /
bfp2_b in hardware
. Block-float is one shared 8-bit exponent per 16 values (a face row), with
7/3/1-bit magnitude plus sign, which is 1.0625 / 0.5625 / 0.3125 B per element. Those are the only
sub-8-bit formats Blackhole can stream at line rate. There is no codebook or LUT unpack mode, and no
MXFP4/NVFP4 (those are Quasar-only; docs/QUANTIZATION.md). A weight tile laid out [K, N]
(tp_common.shard_w transposes to [in, out]) shares exponents along N, across 16 output
channels at the same input index.

Where a trellis decode could run

unit what it can do cost for 12.2 B weights per card per step (ESTIMATE)
SFPU (vector unit, 32 lanes/Tensix) int add/shift/and/or/xor; SFPMUL24 (23x23-bit, Blackhole-only) 16x32-bit multiply mod 2^32 needs about 3 SFPMUL24 + adds/shifts. Byte-sum via SWAR is about 6 ops. Extract about 4 ops, convert+FMA about 2, plus load/store. ~15-20 ops per 32-weight row, about 10-20 cycles with SFPLOADMACRO overlap. Time = 2.56 ms x cycles/row, so ~25-50 ms per step
FPU (matrix engine) tile matmul from SrcA/SrcB only cannot do integer bit ops
Reader RISC-V (BRISC/NCRISC, scalar RV32IM) anything, scalar ~12 instructions per weight on 220 cores, ~0.7 s per step: dead on arrival
Unpacker bfp/fp formats only no trellis/LUT mode

Reference points for the SFPU estimate (THIRD-PARTY): a full 32x32-bit integer multiply on Blackhole
is 8 cycles per 32-value row even hand-scheduled with SFPLOADMACRO (jasondavies.com, 2025-11).
Our multiply is cheaper (16-bit operand), but the byte-sum has no dp4a equivalent. Two structural
costs come on top. First, decoded values leave through the packer to L1 and come back through the
unpacker into SrcB for the FPU, which is an extra round trip per tile. Second, SFPU and FPU are
driven by the same math thread, so decode and matmul largely serialize within a core.

Break-even (ESTIMATE): production reads 9.33 GB/card/token, which is 20.6 ms at 454 GB/s. 4-bpw EXL3
reads 6.57 GB/card, 14.5 ms. With a serial decode, EXL3 wins only if decode takes under ~4-6 ms,
which means <=1.6-2.4 cycles per 32-weight row. With decode perfectly overlapped against DRAM,
it wins only if decode finishes under 14.5 ms, which means <=5.6 cycles per row, and even then
the gain caps at ~6 ms. Neither is credible given an 8-cycle hand-tuned 32-bit multiply alone.

Hadamards on Tensix

An online had128 is a [M*k/128, 128] @ H128 matmul, cheap in FLOPs. But EXL3 needs one in and
one out per linear, plus one copy per fused constituent because each has its own suh. That is
~400-640 extra traced ops per decode step. At ~5-15 µs device time each, that is ~2-9 ms
(ESTIMATE). It has to be fused into the matmul reader/epilogue or folded offline to be free. For
TP=2 the blocks divide cleanly: row-parallel K shards are 8704 = 68x128 and 3072 = 24x128, and
because the Hadamard is linear, the output one can be applied before or after the all-reduce. Only a
residual-stream rotation (QuaRot-style) can be folded into weights at zero runtime cost. It makes
the output (N) dim of o_proj/out_proj/down_proj incoherent, which is the axis Tensix bfp
exponents are shared along. For qkv/in_proj/gate_up it rotates K, which is the wrong axis for
bfp blocks (hypothesis, testable offline).

Prior art on Tenstorrent

  • Trellis/QTIP/EXL3/codebook dequant on Tenstorrent: none found. Searched tt-metal issues and PRs
    and tenstorrent org PRs for trellis, QTIP, exllama, exl3, codebook and "lookup table dequant";
    zero hits.
  • The closest native analogue exists: MatmulCustomCompressed plus CompressedTensor in
    models/demos/deepseek_v3_b1/, with per-32x32-tile format codes {bfp8, bfp4, bfp2, zero} driven by
    BitSculpt "BSPM" precision maps (compressed_tensor/bspm_loader.py; issues #38520, #39989).
    THIRD-PARTY: DeepSeek experts at 3.5 bits/element (78% bfp4, 22% zero) passed Lite PPL 2.98 and
    AIME24 85.4%, and failed Full PPL by +0.08 (#42601). Open defects: wrong PCC (~0.04) at 32 cores
    with any non-bfp8 tile
    (#42841) and a hang with zero bfp8 tiles (#42586).
  • tt-metal #57410 (open, 2026-09-22): ttnn.from_torch(..., bfloat4_b) (the host packer, which
    is how our bf4 weights are made) rounds straight to 3 mantissa bits (RNE). The device typecast
    rounds to BFP8 first and then truncates. They differ on 42.5% of elements. The unpack side is
    correct. Two consequences for us. First, the host is free to write any bits we choose, which
    is what makes E1's "grid-snapped checkpoint" work. Second, any on-device re-quantization (e.g. of
    KV or activations) does not match host-made weights bit for bit.

Options compared

"Δ decode" is against production (all-MLP bf4, 9.33 GB/card/token). The two numbers are the
measured in-model slope (0.78 ms/GiB saved, from goal3-down-proj-bf4-2026-09-05.jsonl, taken
on an older stack) and the bandwidth-ideal figure (454 GB/s). Both are ESTIMATES.

option runtime change Δ decode ms/step Δ card memory quality vs prod effort verdict
A Load-time transcode EXL3 to bf8 (all linears) none (new loader) +2.1 to +3.1 (slower), since it is the goal1 byte count -2.66 GiB ~EXL3-4bpw everywhere: better on MLP, worse on attn/GDN (4-bit source vs FP8) 1-2 wk compatibility demo only
A' Transcode EXL3 to bf4 none -2.6 to -4.0 +3.37 GiB quantized twice, worse than prod on attn/GDN 1-2 wk no
B On-device trellis decode (SFPU) + FPU matmul new compute kernel + had128 in/out +10 to +45 (slower); the -4 to -6 ceiling needs decode for free +5.16 GiB (4 bpw), +7.99 GiB (3 bpw) EXL3-exact 2-3 months, research-grade NO-GO unless the microprobe shows <=2 cycles/row
C Dequant in reader RISC-V new DM kernel ~+700 as B EXL3-exact months NO-GO
D Store Hadamard-domain W_inner as bfp + online had ~400-640 extra ops +2 to +9 (had ops) +/- bytes as chosen better than RTN bfp at equal bytes 3-4 wk no, unless hads fuse
E0 Accuracy gate for today's bf4 weights none 0 0 measures prod 3-5 d do (needed by everything)
E1 GPTQ/LDLQ to bfp4_b grid-snapped bf16 checkpoint (MLP first, then attn/GDN) none 0 (MLP only); -2.6 to -4.0 if attn/GDN go bf4 0 / +3.37 GiB should beat RTN bfp4 at equal bytes (hypothesis) 2-3 wk do, if E0 shows headroom
E2 E1 + folded residual-stream rotation none (weights + norms rewritten) same as E1 same further gain on o/out/down (hypothesis) +1-2 wk after E1
E3 Per-tile mixed bfp8/4/2/0 (MatmulCustomCompressed + BSPM) swap decode matmul kernel on TP=2 DRAM-sharded path -3 to -6 (at ~3.5 b/e) +3 to +5 GiB sensitivity-optimal per tile 4-8 wk + upstream bugs only if E1 fails on some tensors

Byte model (from bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonl element counts, streamed
per token, embed excluded): goal1 21.52 GB total / 10.76 per card; prod 18.67 / 9.33; all-linear
bf4 with lm_head bf8 15.05 / 7.52; EXL3 4 bpw (head 6) 13.13 / 6.57; EXL3 3 bpw 10.09 / 5.04.

Does it pay here? (with numbers)

Makespan model. Each cell of the 2026-09-14 sweep satisfies makespan ≈ TTFT_max + 1024 ×
decode_ms
to within 1.3 s (MEASURED inputs, sweep-32k-20260914T142154Z.table.txt; e.g. B=8/32k:
80.84 + 67.80 = 148.6 vs 148.72 measured). A per-step saving Δ therefore moves makespan by
1.024·Δ seconds. Decode's share of makespan is 88% (B=4/4k) down to 46% (B=8/32k). (This
contradicts the "prefill is 74-79% of wall" wording in CLAUDE.md 0aaaaaaaaaaaaaaaaa: by this
identity it is decode that is ~72-78% of wall at B=8/4k-8k.)

shape decode share E1 attn/GDN→bf4 (-2.6..-4.0 ms) free-decode EXL3 4 bpw (-4.0..-6.1 ms) realistic B (+10..+30 ms)
B=4 / 4k 88% -4.3% .. -6.7% -6.7% .. -10.2% +17% .. +50%
B=4 / 32k 60% -2.7% .. -4.2% -4.2% .. -6.3% +10% .. +31%
B=8 / 4k 78% -3.6% .. -5.6% -5.6% .. -8.5% +14% .. +42%
B=8 / 16k 60% -2.6% .. -4.0% -4.0% .. -6.0% +10% .. +30%
B=8 / 32k 46% -1.8% .. -2.8% -2.8% .. -4.2% +7% .. +21%
8x128k (TTFT_max 443.2 s) ~15% ~-0.5% .. -0.8% -0.8% .. -1.2% +2% .. +6%
4x256k (TTFT_max 609.2 s) ~12% ~-0.4% .. -0.6% -0.6% .. -0.9% +1.5% .. +4.5%

(Target-shape rows assume 1024 output tokens at 70-90 ms/step. That decode rate is an ESTIMATE; it
has not been measured at 128k/256k.)

Caveats that must travel with these numbers:

  • Standalone byte savings have not converted 1:1 in-model here. The down_proj bf4 A/B
    delivered 3.0% against a predicted 15.2% (goal3-down-proj-bf4-2026-09-05.jsonl), and N1
    records a 2.35x faster standalone down_proj that ran 0-2% slower in-model. The measured-slope
    column already discounts for this. The bandwidth column does not.
  • The sweep arms are goal1 (down bf8). Production's down bf4 is already worth ~2.1-3.1 ms/step
    relative to those sweep cells. Re-baseline on the prod weight mix before crediting any new
    weight work.
  • Prefill: bf4 vs bf8 weights made 0% difference at M=512 (docs/QUANTIZATION.md,
    bench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl), since prefill is compute-bound. So
    no option here touches TTFT, which dominates at the target shapes. For B, prefill would get
    worse: a reconstruct per chunk plus bf16/HiFi matmuls where today runs LoFi.
  • With prefix caching now live (commit 4e91d87), cached heads shrink TTFT, which raises decode's
    share and makes E1's saving worth somewhat more on repeated-prefix agent workloads.

Memory angle. The budget is 61.73 GiB usable. Weights take ~25.1 GiB, the target KV is 32 GiB at
bf8, trace 0.5-1 GiB, and margin is ~1.9-3.6 GiB (docs/GOAL-AGENT-SERVING.md, prod profile
comment). At bf8 KV, 1 GiB ≈ 32,768 tokens across the pair.

weight mix Δ vs prod extra bf8 KV tokens what it buys (batch is capped at 8)
E1: attn/GDN→bf4, lm_head bf8 +3.37 GiB ~110k a comfortable margin at 4x256k/8x128k, plus room for prefix-cache blocks and GDN snapshots beyond the live batch
EXL3 4 bpw (hypothetical B) +5.16 GiB ~169k same, more
EXL3 3 bpw (hypothetical B) +7.99 GiB ~262k about one extra 256k context's worth. Not usable as a 5th agent (B cap)

Batch is capped at 8 by owner directive, so capacity does not buy "more agents". It buys margin
(two host freezes on 2026-09-14 came from memory pressure, albeit host-side) and prefix-cache
residency
, which is the enabling feature at the target shapes.

Integration plan (phases, probes, gates)

Principles: every probe fails closed and writes a bench/runs/*.jsonl record even on failure.
Every instrument is exercised on fixtures that force pass, fail and null before it sees real data.
Arms self-identify from server banners, and the weight arm must log per-group dtype plus the
checkpoint/tensor-cache hash, like patches/tt-metal/0016. The trap to avoid: ttnn.as_tensor
reloads a cache file as-is, so a stale tensor_cache_* dir silently serves the old weights. A new
quantization must use a distinct cache suffix. Arms are interleaved on one source tree and
differ only by environment or checkpoint path.

P0: offline error study (desk + off-box GPU, no cards). 3-5 days

  • Collect per-linear calibration Hessians for Qwen3.8-27B bf16 on a CUDA box (not cfx-llm2,
    whose probe budget is ~5.5 GB of host RAM). The exllamav3 converter already does this, and its
    sc_trace.py self-sampled trace matches our agentic/reasoning distribution better than web text.
  • For each linear group (gate/up, down, attn qkv/o, GDN qkvzab/out, lm_head), compute proxy error
    tr(E H E^T)/tr(W H W^T) for:
    (i) RTN bfp4_b using an exact emulation of the tt-metal host packer
    (blockfloat_common.cpp::convert_u32_to_bfp, per #57410), from both the FP8 and the bf16 source;
    (ii) RTN bfp8_b;
    (iii) GPTQ/LDLQ onto the bfp4_b grid;
    (iv) (iii) + residual rotation;
    (v) EXL3 4.0/3.0 bpw, reconstructed with exl3xpu/ref.py (bit-exact).
  • Go to P2/P3 if (iii) cuts proxy error by >=2x vs (i) on attn/GDN tensors and brings it within
    2x of (v)-4bpw. No-go (stop E1) if (iii) gains <25% over (i) everywhere.

P1 (E0): accuracy gate, built once and reused. 3-5 days + 2 card holds

GSM8K on short prompts is not acceptable here: it is near-blind to KV dtype, and the same problem
applies to weights through the long GDN recurrence. Proposed gate, three layers:

  1. Teacher-forced logit gate (exl3xpu Gate A3 pattern). A sealed panel of N windows. Report
    KL(ref‖arm), top-1 agreement and max-KL at positions deep in long contexts: the last 256
    tokens of 4k, 32k and 128k documents. Quantization error in in_proj_qkvz compounds through 48
    recurrent states, and a 256-token panel cannot see that. The reference is bf16 HF on an off-box
    GPU. The device side uses vLLM prompt_logprobs (probe first that the TT plugin supports it; the
    host-sampling route that --logprobs forces is fine for accuracy, just not for the argmax path).
    Pre-register thresholds relative to the current prod arm: a new weight arm must be <= prod
    KL at every context.
  2. Task gate at the owner's workload. Harder keyed retrieval: more needles, dense decoys,
    multi-hop, at 16k/32k/64k (bench/kv-accuracy.py; the 2026-09-13 Gate A sat at its ceiling).
    Add the 20-turn agentic tool-call canary (phase2a-prefix-e2e correctness stage) and a
    code-generation subset with thinking on. Report the rule-of-three bound, never "equal".
  3. Reproducibility. A sha self-check at fixed width, at B=4 and B=8 (width changes greedy
    output). A format change legitimately changes outputs, so sha-identity to the old arm is not
    the bar.
  • First deliverable: the missing accuracy record for production's bf4 MLP (RTN from FP8)
    against bf16 and against goal1 (down bf8).

P2 (E1a): grid-snapped MLP-only checkpoint. 1-2 weeks + 2-3 card holds

  • The quantizer writes bf16 safetensors whose values lie exactly on the bfp4_b grid. It simulates
    the packer's shared-exponent choice, including the case where rounding the block max crosses a
    binade. Every bfp4_b value is exactly representable in bf16, so the host packer is the
    identity
    and no loader or kernel change is needed. Verify by round trip: host-pack, unpack,
    compare 100% bitwise on every tensor, offline with ttnn host APIs and no device.
  • Same bytes as prod, so predicted Δ decode 0 ± 0.5 ms. A shift beyond that means the arm is
    contaminated. Gate: P1 KL strictly below prod RTN at every context. Discard if not.
  • Nix: package the quantizer as nix run .#bfp-quant (off-box CUDA, per the one-stop-shop
    directive) and pin the output checkpoint by hash like any other model input.

P3 (E1b): attention/GDN projections to bf4. 1-2 weeks + 3-4 card holds

  • Needs a dtype gate per group (extend QWEN36_MLP_DOWN_BF4 to a small policy env var, banner-logged).
  • Pre-registered prediction: -2.6 to -4.0 ms/step at B=4 and B=8 (weight read is batch-flat).
    Falsifier: < 1.0 ms. Measured with bench/ab/sweep-32k.sh extended with a prod-weight-mix
    control arm (interleaved), reported as makespan and TTLT, never decode ms alone.
  • Gate: P1 KL <= prod at every context, and retrieval/agentic within the rule-of-three bound.
    Go to default only if both hold and makespan improves beyond the 0.33% noise floor at >= 3 of
    4 contexts.

P4 (E2/E3, optional). 2-8 weeks

  • E2 (residual rotation) only if P0 shows (iv) clearly better than (iii) on o/out/down. It needs
    norm-weight folding checked against Qwen3.5's RMSNorm parameterization, and GDN's in_proj_a/b
    (bf16) rotated consistently.
  • E3 (per-tile mixed precision) only if some tensor group fails P3's gate at bf4 but passes at
    bf8. Prerequisites: #42841 (32-core PCC) and #42586 (hang) fixed upstream, then port
    MatmulCustomCompressed onto our TP=2 DRAM-sharded 1D decode progcfgs. BSPM maps would come
    from a sensitivity pass (exllamav3 sc_optimize.py is the model for this).

P5 (A, optional, owner's call): "runs EXL3 checkpoints" compatibility. ~1 week

  • Add is_exl3_checkpoint next to the existing is_fp8_checkpoint / is_nvfp4_checkpoint in
    weight_mapping.py, called from model_config.load_state_dict. Reconstruct with a vendored
    ref.py (MIT) to the original basis, then shard_w to bf8 (quality-preserving) or bf4.
  • Trap: ref.weight_orig builds a full fp32 k x n. For lm_head [248320, 5120] that is
    5.1 GB, which is an instant SIGKILL on cfx-llm2 (same trap as mtp_head_oracle). Reconstruct in
    column chunks. Transcode once off-box into the tensor cache rather than on the serving host.
  • Gate: reconstructed weights bit-identical to exllamav3.reconstruct() (the exl3xpu Gate A1
    approach) plus P1. Expect slower decode than prod (bf8) or worse attn/GDN quality (bf4).

P-B (only if someone insists on on-device trellis). Half a day of card time

  • A single-core SFPU microkernel for the K=4 mul1 decode (extract, SFPMUL24 multiply, SWAR
    byte-sum, cast, FMA, store) over L1-resident words. It reads the cycle counter and records
    cycles per 32-weight row.
  • Go only if <= 2.0 cycles/row (serial break-even). Between 2 and 5.6 would need proven
    decode/DRAM overlap and FPU concurrency, which is a second probe. Above 5.6: close permanently,
    in writing. Expected (ESTIMATE): 10-20 cycles/row, so close.

Effort total for the recommended path (P0-P3): ~5-8 engineer-weeks and ~8-10 card holds,
every one serialized on /tmp/ttlock with the MemAvailable >= 4 GB check.

Risks / unknowns

  • Conversion of bytes to ms is weak here (3.0% delivered vs 15.2% predicted). E1b's -2.6 to
    -4.0 ms could land near zero. The falsifier is set at 1.0 ms.
  • bf4 attention/GDN may not pass any reasonable gate. unsloth's NVFP4 recipe keeps
    attn/linear_attn/lm_head and the last 8 MLP layers at 8-bit; QUASAR gets all-4-bit only via QAT
    (docs/QUANTIZATION.md). GPTQ-style rounding narrows but may not close that gap.
  • The recurrent state compounds error. 48 GDN layers carry state across up to 256k tokens.
    Short-panel metrics can pass while long contexts drift. That is why P1 measures deep positions.
  • The bfp exponent axis is along N ([K,N] tile layout). Rotations help only when applied along
    N. Verify the actual on-device layout of each weight (some matmul configs may transpose) before
    designing E2.
  • Host packer semantics may change upstream (#57410 proposes aligning the host to the device).
    The grid-snapped checkpoint stays safe only if values are exactly representable, so the packer
    does not round at all. The round-trip test in P2 must re-run on every tt-metal pin bump.
  • Upstream compressed-matmul bugs (#42841, #42586) block E3 on exactly our core counts (32/33/44).
  • Off-box compute is needed for Hessians and quantization (27B, ~2 h per full conversion on one
    fast GPU per exllamav3 doc/optimize.md, THIRD-PARTY). cfx-llm2 cannot host it.
  • Side observation, not a proposal: exl3xpu runs vLLM's batched MTP (qwen3_5_mtp, k=3) on this
    exact model with acceptance ~2.4-3.1 at C=1..16 and 2-2.4x aggregate at C=8 (THIRD-PARTY). Owner
    directive 0aaaaa dropped MTP, and "do not propose speculation" stands. This is recorded only
    because the directive says MTP was never measured here and must never be recorded as rejected
    on evidence.

Sources

This repo (MEASURED / internal):

  • bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl: per-projection read/non-read split, 454-469 GB/s, bytes/element 0.5625/1.0625
  • bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonl: per-group element counts (basis of all byte arithmetic)
  • bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl: down_proj bf4 in-model: +3.1% vs +15.2% predicted, accuracy unverified
  • bench/runs/sweep-32k-20260914T142154Z.jsonl and .table.txt: makespan/TTFT/decode, goal1 profile (no down bf4)
  • bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl, ...B4-256k-20260914T134045Z.jsonl (CLAUDE.md cites a non-existent ...134735Z): 443.2 s / 609.2 s
  • bench/runs/gateA-kv-accuracy-20260913T215707Z.jsonl: retrieval gate design and ceiling problem
  • bench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl: bf4 vs bf8 at LoFi (0% at prefill M=512; PCC 0.9932 vs 0.9999)
  • scripts/vllm-tt-serve.sh:155 (prod sets QWEN36_MLP_DOWN_BF4=1); bench/ab/sweep-32k.sh:57,82 (arms lack it)
  • patches/prod-local-metal/models/demos/blackhole/qwen36/tt/{model_config.py,tp_common.py,model.py}: loader (load_state_dict, is_fp8/nvfp4_checkpoint), shard_w [in,out] layout, lm_head bf8
  • docs/QUANTIZATION.md, docs/VRAM-BUDGET.md, docs/SATURATION-AND-SRAM.md, docs/GOAL-AGENT-SERVING.md, docs/PERFORMANCE.md

THIRD-PARTY:

> Desk research only. No tt-metal/vLLM process was started and the cards were not touched. > Labels used throughout: **MEASURED** (cites a `bench/runs/*` record in this repo), **THIRD-PARTY** > (someone else's number, not reproduced here), **ESTIMATE** (arithmetic done for this issue; not a result). > Standing rule: arithmetic on a measured slope is not a result. The one in-model A/B of a predicted > saving delivered 0 to -2%, and the down_proj bf4 A/B delivered 3.0% against a predicted 15.2%. ## TL;DR verdict **Porting exl3xpu as it stands is a NO-GO. It would make decode slower, not faster. There are two cheaper things worth doing instead, and both use EXL3's ideas while leaving its bitstream behind.** 1. **exl3xpu is not portable code.** It is an Intel Arc B70 (Xe2, "XPU" = PyTorch's Intel GPU device) vLLM plugin. Its kernels are SYCL/ESIMD built around Xe2 `dpas` (VNNI) and `dp4a`. It borrows GDN and attention from vLLM-XPU and has no tensor parallelism. Nothing in it runs on Tensix. What carries over is the *format spec*, the bit-exact PyTorch reference decoder (`exl3xpu/ref.py`, MIT), and the accuracy-gate method. 2. **Decoding the trellis on the card costs more than it saves (ESTIMATE, one cheap probe away from MEASURED).** Per card, each decode step would have to turn about 12.2 B trellis weights into tiles. On Blackhole that work lands on the SFPU (32 lanes per Tensix; `SFPMUL24` is the only integer multiply, and it is 23-bit). Every weight needs a 16-bit window extract, a 16x32-bit multiply mod 2^32, a byte-sum and one fp FMA, which is roughly 15-20 SFPU ops per 32-weight row. That puts the decode at **~25-50 ms per step**. The most 4-bpw EXL3 could save against production's weight read is **~4-6 ms per step**, and only if the decode were free. Break-even needs **<=2 SFPU cycles per 32-weight row** if decode runs serially, or <=5.6 if it overlaps the DRAM read perfectly. Doing the dequant in the RISC-V reader kernels is about 50x worse (~0.7 s per token). 3. **Even a free 4-bpw EXL3 barely moves the owner's metric.** Weight read is ~37% of the decode step. The bytes EXL3 would save relative to *production* (which already runs all MLP at bf4) are worth **-4 to -6 ms per step**. That is **-4% to -10% makespan on the <=32k sweep** and **~-1% at 8x128k / 4x256k**, where serialised prefill (443 s / 609 s) dominates. For comparison, prefix caching and prefill work each move those shapes by tens of percent. 4. **What is worth doing, ranked:** - **(E0) An accuracy gate for the bf4 weights we already ship.** Production runs `mlp.down_proj` at bf4 (`QWEN36_MLP_DOWN_BF4=1` in the `prod` profile). Its only record says *"accuracy: unverified against a bf8 baseline"* (`bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl`). This costs nothing at runtime and is needed before any weight-format change. - **(E1) An "EXL3-inspired" offline quantizer that writes native `bfp4_b`.** It rounds with Hessian awareness (GPTQ/LDLQ, EXL3's error-feedback step) straight onto the bfp4_b grid and emits a *grid-snapped bf16 checkpoint*. The host packer reproduces those bits exactly, so there is **zero runtime change**. Value: better bf4 quality where we already use it, and possibly enough quality to move attention/GDN projections from bf8 to bf4. That second step is worth about **-2.6 to -4.0 ms per step and +3.4 GiB of card memory (~110k bf8 KV tokens)** (ESTIMATE). - **(E3, only if E1 falls short on some tensors) Per-tile mixed bfp8/bfp4/bfp2/zero.** This uses Tenstorrent's own `MatmulCustomCompressed` plus BitSculpt "BSPM" precision maps. It is the native equivalent of EXL3's per-tensor bitrate recipes. It exists in tt-metal (`deepseek_v3_b1`) but has open correctness bugs at 32 cores. 5. **Running turboderp's EXL3 checkpoints on our cards is possible (option A: load-time transcode), but it is a compatibility feature, not a performance one.** Transcoding EXL3 to bf8 keeps EXL3's quality but reads more bytes than production. Transcoding to bf4 quantizes twice and loses to our current source on attention/GDN. **Correction to the brief:** production is *not* "gate/up bf4, down bf8". The `prod` profile sets `QWEN36_MLP_DOWN_BF4=1` (`scripts/vllm-tt-serve.sh:155`), so production streams **~9.33 GB/card/token**. The 10.76 GB/card figure belongs to the `goal1` profile, which is what the benchmark drivers use (`bench/ab/sweep-32k.sh`, `decode-serving-ab.sh`). Consequence: **the 2026-09-14 "clean-slate" sweep's optimized arm is not production's weight mix.** Its record's `env_base` has no `QWEN36_MLP_DOWN_BF4`. --- ## What exl3xpu is - Repo: `github.com/0xSero/exl3xpu` @ `2d17c57` (2026-09-24), MIT. It is a vLLM 0.26.1 **XPU** plugin (`vllm.general_plugins` entry point, `register_quantization_config("exl3")`). - Hardware: **Intel Arc Pro B70** (BMG-G31, Xe2, 32 GB). "xpu" is PyTorch/oneAPI's Intel GPU device, not a generic accelerator. Kernels are SYCL **ESIMD** (`csrc/exl3_esimd.h`, 767 lines), JIT-compiled with icpx (`spir64`). - It consumes **turboderp's EXL3** checkpoints (exllamav3 v1.x format). The served model is **exactly ours**: `turboderp/Qwen3.8-27B-exl3` @ 4.00 bpw (lm_head 6 bpw, `mul1` codebook, 401 EXL3 tensors, 14.91 GiB). `in_proj_a/b`, norms, embeddings and the vision tower stay bf16. - Scope: vLLM supplies scheduling, paged KV, and the **GDN and attention kernels** (the XPU ones). exl3xpu only replaces the linear layers. **No tensor parallelism** ("use data parallel"). Only 4 and 6 bpw `mul1` are enabled by default; 2/3/5 bpw and the mcg/3INST codebooks need `-DEXL3_ALL_CODEBOOKS`. - Correctness: dequantized weights are **bit-identical** to exllamav3 `reconstruct()` for all 401 tensors on every kernel path. Logits vs exllamav3 on an RTX 3090: **top-1 99.63%, KL 9.8e-5** on a sealed 64x256 teacher-forced panel ("Gate A3"). This is *implementation* equivalence against EXL3 itself, not quantization quality against bf16. - Performance, THIRD-PARTY (one B70, `bench/results/2026-09-23.jsonl`, `recipe.json`): | | exl3xpu, 1x B70 | ours, 2x P150a (MEASURED) | |---|---|---| | prefill 4k / 32k / 128k / 254k | 1589 / 1497 / 1049 / 763 tok/s | 3,534 tok/s @16k, 2,366 @128k, 1,707 @256k | | decode C=8 aggregate, no MTP | 152 tok/s (`docs/GOAL.md` T4 floor) | ~142.6 tok/s decode-window (B=8/4k, 56.10 ms, sweep optimized arm) | | decode C=8 aggregate, MTP k=3 (thinking on) | 294-363 tok/s | n/a (MTP dropped by owner) | | linears per step at M=1 / M=4 | 27.3 / 31.7 ms (after K=6 planar fix) | weight read 23.34 ms of step (N1) | The harnesses and definitions differ, so this is not an A/B. Our records: `bench/runs/serving-16k-20260913T095210Z.jsonl`, `bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl`, `bench/runs/target-shape-ttft-B4-256k-20260914T134045Z.jsonl`, `bench/runs/sweep-32k-20260914T142154Z.jsonl`. - Its own diagnosis is relevant here: on Xe2 the M=1 GEMV is **co-limited by integer-ALU issue (~22-25 ms) and DRAM (~23.5 ms)**. The trellis decode sits "near the integer-ALU roofline", on a GPU with far more integer throughput per weight than a Tensix SFPU (see below). ## EXL3 format & kernel anatomy EXL3 is turboderp's implementation of **QTIP** (Tseng et al., NeurIPS 2024: *Quantization with Trellises and Incoherence Processing*). The quantizer has three components, and only one of them is tied to the bitstream: 1. **Incoherence processing.** `W = diag(suh) · H · W_inner · H · diag(svh)`, where `H` is the normalized 128x128 Sylvester Hadamard applied blockwise and `suh`/`svh` are fp16 sign/scale vectors. At inference: `y = had128(had128(x * suh) @ W_inner) * svh`. This means an input Hadamard and an output Hadamard **per linear**. 2. **Hessian-aware rounding (LDLQ).** Calibration activations give `H = X^T X`, which is regularized and LDL-decomposed. Tiles are quantized in order with error feedback (`exllamav3/modules/quant/exl3_lib/quantize.py:549 ldlq`, `:1471 quantize_exl3`). The converter's proxy error is `tr(E H E^T) / tr(W H W^T)`. 3. **Trellis-coded quantization with a computed codebook.** Each 16x16 tile is `256*K` bits (K = bpw, 1..8). Value `t` is the **16-bit window ending at bit `(t+1)*K`**, circular within the tile, placed at a tensor-core permutation. The `mul1` codebook, bit-exact per `ref.py`: `x = state * 0x83DCD12D (mod 2^32)`, then `v = fp16(1024 + bytesum(x)) * fp16(0x1EEE) + fp16(0xC931)`, one fp16 FMA with a single rounding. Per weight, the inner loop is: funnel-shift two 32-bit words, mask to 16 bits, 32-bit integer multiply, sum 4 bytes, int to fp16, one FMA. The Xe2 kernel does this in about **9 SIMD32 instructions per 32 values**: `shl/shr/or/mul/dp4a/mov/mad`. `dp4a(0x6400, x, 0x01010101)` does the byte-sum and fp16 bias in one instruction, and that trick is the reason it is fast. Memory access is a pure contiguous stream of trellis words per tile row (K=4 packs 8 values per word). The 128-point Hadamards are separate small kernels (~1.2 ms per step under graphs). For M>128 (prefill) the plugin reconstructs fp16 `W_inner` slices and calls oneDNN GEMM. Available bitrates: 1-8 bpw per tensor, with per-tensor recipes from `sc_optimize.py`. THIRD-PARTY quality, exllamav3 `doc/llama31_8b_instruct_kld_bpw.png` (Llama-3.1-8B, KL vs bf16): **EXL3 3.0 bpw 0.053, 3.5 bpw 0.030, 4.0 bpw 0.013, 5.0 bpw 0.003**; GGUF Q4_K_M 0.018, Q6_K 0.003. Nobody has measured KL for RTN `bfp4_b` on our model. That is the gap E0/E1 close. Qwen3.x hybrid/GDN support: **yes**. exllamav3 quantizes the GDN projections (`in_proj_qkvz`, `out_proj`) like any linear, and exl3xpu serves the 48-GDN/16-attention model through vLLM's GDN kernels. Nothing in the format is GDN-specific. ## Blackhole mapping Hardware facts (MEASURED or from source, `docs/SATURATION-AND-SRAM.md`): p150a is 2x-harvested, **110 compute Tensix** (a column is reserved for dispatch), **1.5 MiB L1 each** (165 MiB per card), AICLK 1350 MHz, 8 GDDR6 channels. Usable DRAM is **30.87 GiB per card** (`docs/VRAM-BUDGET.md`). Decode matmul readers reach **454-469 GB/s** (`bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl`). What the Tensix pipeline gives us for free: the **unpacker decompresses `bfp8_b` / `bfp4_b` / `bfp2_b` in hardware**. Block-float is one shared 8-bit exponent per 16 values (a face row), with 7/3/1-bit magnitude plus sign, which is 1.0625 / 0.5625 / 0.3125 B per element. Those are the only sub-8-bit formats Blackhole can stream at line rate. There is no codebook or LUT unpack mode, and no MXFP4/NVFP4 (those are Quasar-only; `docs/QUANTIZATION.md`). A weight tile laid out `[K, N]` (`tp_common.shard_w` transposes to `[in, out]`) shares exponents along **N**, across 16 output channels at the same input index. ### Where a trellis decode could run | unit | what it can do | cost for 12.2 B weights per card per step (ESTIMATE) | |---|---|---| | **SFPU** (vector unit, 32 lanes/Tensix) | int add/shift/and/or/xor; **`SFPMUL24`** (23x23-bit, Blackhole-only) | 16x32-bit multiply mod 2^32 needs about 3 `SFPMUL24` + adds/shifts. Byte-sum via SWAR is about 6 ops. Extract about 4 ops, convert+FMA about 2, plus load/store. **~15-20 ops per 32-weight row**, about 10-20 cycles with `SFPLOADMACRO` overlap. Time = 2.56 ms x cycles/row, so **~25-50 ms per step** | | **FPU** (matrix engine) | tile matmul from SrcA/SrcB only | cannot do integer bit ops | | **Reader RISC-V** (BRISC/NCRISC, scalar RV32IM) | anything, scalar | ~12 instructions per weight on 220 cores, **~0.7 s per step**: dead on arrival | | **Unpacker** | bfp/fp formats only | no trellis/LUT mode | Reference points for the SFPU estimate (THIRD-PARTY): a full 32x32-bit integer multiply on Blackhole is **8 cycles per 32-value row** even hand-scheduled with `SFPLOADMACRO` (jasondavies.com, 2025-11). Our multiply is cheaper (16-bit operand), but the byte-sum has no `dp4a` equivalent. Two structural costs come on top. First, decoded values leave through the **packer** to L1 and come back through the **unpacker** into SrcB for the FPU, which is an extra round trip per tile. Second, SFPU and FPU are driven by the same math thread, so decode and matmul largely **serialize** within a core. Break-even (ESTIMATE): production reads 9.33 GB/card/token, which is 20.6 ms at 454 GB/s. 4-bpw EXL3 reads 6.57 GB/card, 14.5 ms. With a serial decode, EXL3 wins only if decode takes under ~4-6 ms, which means **<=1.6-2.4 cycles per 32-weight row**. With decode perfectly overlapped against DRAM, it wins only if decode finishes under 14.5 ms, which means **<=5.6 cycles per row**, and even then the gain caps at ~6 ms. Neither is credible given an 8-cycle hand-tuned 32-bit multiply alone. ### Hadamards on Tensix An online `had128` is a `[M*k/128, 128] @ H128` matmul, cheap in FLOPs. But EXL3 needs one in and one out **per linear**, plus one copy per fused constituent because each has its own `suh`. That is ~400-640 extra traced ops per decode step. At ~5-15 µs device time each, that is **~2-9 ms** (ESTIMATE). It has to be fused into the matmul reader/epilogue or folded offline to be free. For TP=2 the blocks divide cleanly: row-parallel K shards are 8704 = 68x128 and 3072 = 24x128, and because the Hadamard is linear, the output one can be applied before or after the all-reduce. Only a **residual-stream** rotation (QuaRot-style) can be folded into weights at zero runtime cost. It makes the *output* (N) dim of `o_proj`/`out_proj`/`down_proj` incoherent, which is the axis Tensix bfp exponents are shared along. For `qkv`/`in_proj`/`gate_up` it rotates K, which is the wrong axis for bfp blocks (hypothesis, testable offline). ### Prior art on Tenstorrent - **Trellis/QTIP/EXL3/codebook dequant on Tenstorrent: none found.** Searched tt-metal issues and PRs and tenstorrent org PRs for trellis, QTIP, exllama, exl3, codebook and "lookup table dequant"; zero hits. - **The closest native analogue exists:** `MatmulCustomCompressed` plus `CompressedTensor` in `models/demos/deepseek_v3_b1/`, with per-32x32-tile format codes {bfp8, bfp4, bfp2, zero} driven by **BitSculpt "BSPM" precision maps** (`compressed_tensor/bspm_loader.py`; issues #38520, #39989). THIRD-PARTY: DeepSeek experts at 3.5 bits/element (78% bfp4, 22% zero) passed Lite PPL 2.98 and AIME24 85.4%, and failed Full PPL by +0.08 (#42601). Open defects: **wrong PCC (~0.04) at 32 cores with any non-bfp8 tile** (#42841) and a **hang with zero bfp8 tiles** (#42586). - **tt-metal #57410 (open, 2026-09-22):** `ttnn.from_torch(..., bfloat4_b)` (the host packer, which is how our bf4 weights are made) rounds straight to 3 mantissa bits (RNE). The device `typecast` rounds to BFP8 first and then truncates. They differ on 42.5% of elements. The unpack side is correct. Two consequences for us. First, **the host is free to write any bits we choose**, which is what makes E1's "grid-snapped checkpoint" work. Second, any on-device re-quantization (e.g. of KV or activations) does not match host-made weights bit for bit. ## Options compared "Δ decode" is against **production** (all-MLP bf4, 9.33 GB/card/token). The two numbers are the **measured in-model slope** (0.78 ms/GiB saved, from `goal3-down-proj-bf4-2026-09-05.jsonl`, taken on an older stack) and the **bandwidth-ideal** figure (454 GB/s). Both are ESTIMATES. | | option | runtime change | Δ decode ms/step | Δ card memory | quality vs prod | effort | verdict | |---|---|---|---|---|---|---|---| | **A** | Load-time transcode EXL3 to bf8 (all linears) | none (new loader) | **+2.1 to +3.1 (slower)**, since it is the goal1 byte count | -2.66 GiB | ~EXL3-4bpw everywhere: better on MLP, worse on attn/GDN (4-bit source vs FP8) | 1-2 wk | compatibility demo only | | A' | Transcode EXL3 to bf4 | none | -2.6 to -4.0 | +3.37 GiB | quantized twice, worse than prod on attn/GDN | 1-2 wk | no | | **B** | On-device trellis decode (SFPU) + FPU matmul | new compute kernel + `had128` in/out | **+10 to +45 (slower)**; the -4 to -6 ceiling needs decode for free | +5.16 GiB (4 bpw), +7.99 GiB (3 bpw) | EXL3-exact | 2-3 months, research-grade | **NO-GO** unless the microprobe shows <=2 cycles/row | | C | Dequant in reader RISC-V | new DM kernel | ~+700 | as B | EXL3-exact | months | **NO-GO** | | D | Store Hadamard-domain `W_inner` as bfp + online had | ~400-640 extra ops | +2 to +9 (had ops) +/- bytes | as chosen | better than RTN bfp at equal bytes | 3-4 wk | no, unless hads fuse | | **E0** | Accuracy gate for today's bf4 weights | none | 0 | 0 | *measures* prod | 3-5 d | **do** (needed by everything) | | **E1** | GPTQ/LDLQ to bfp4_b grid-snapped bf16 checkpoint (MLP first, then attn/GDN) | **none** | 0 (MLP only); **-2.6 to -4.0** if attn/GDN go bf4 | 0 / **+3.37 GiB** | should beat RTN bfp4 at equal bytes (hypothesis) | 2-3 wk | **do, if E0 shows headroom** | | E2 | E1 + folded residual-stream rotation | none (weights + norms rewritten) | same as E1 | same | further gain on o/out/down (hypothesis) | +1-2 wk | after E1 | | E3 | Per-tile mixed bfp8/4/2/0 (MatmulCustomCompressed + BSPM) | swap decode matmul kernel on TP=2 DRAM-sharded path | -3 to -6 (at ~3.5 b/e) | +3 to +5 GiB | sensitivity-optimal per tile | 4-8 wk + upstream bugs | only if E1 fails on some tensors | Byte model (from `bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonl` element counts, streamed per token, embed excluded): goal1 21.52 GB total / 10.76 per card; **prod 18.67 / 9.33**; all-linear bf4 with lm_head bf8 15.05 / 7.52; EXL3 4 bpw (head 6) 13.13 / 6.57; EXL3 3 bpw 10.09 / 5.04. ## Does it pay here? (with numbers) **Makespan model.** Each cell of the 2026-09-14 sweep satisfies **makespan ≈ TTFT_max + 1024 × decode_ms** to within 1.3 s (MEASURED inputs, `sweep-32k-20260914T142154Z.table.txt`; e.g. B=8/32k: 80.84 + 67.80 = 148.6 vs 148.72 measured). A per-step saving Δ therefore moves makespan by 1.024·Δ seconds. Decode's share of makespan is **88% (B=4/4k) down to 46% (B=8/32k)**. (This contradicts the "prefill is 74-79% of wall" wording in CLAUDE.md `0aaaaaaaaaaaaaaaaa`: by this identity it is *decode* that is ~72-78% of wall at B=8/4k-8k.) | shape | decode share | E1 attn/GDN→bf4 (-2.6..-4.0 ms) | free-decode EXL3 4 bpw (-4.0..-6.1 ms) | realistic B (+10..+30 ms) | |---|---|---|---|---| | B=4 / 4k | 88% | -4.3% .. -6.7% | -6.7% .. -10.2% | +17% .. +50% | | B=4 / 32k | 60% | -2.7% .. -4.2% | -4.2% .. -6.3% | +10% .. +31% | | B=8 / 4k | 78% | -3.6% .. -5.6% | -5.6% .. -8.5% | +14% .. +42% | | B=8 / 16k | 60% | -2.6% .. -4.0% | -4.0% .. -6.0% | +10% .. +30% | | B=8 / 32k | 46% | -1.8% .. -2.8% | -2.8% .. -4.2% | +7% .. +21% | | **8x128k** (TTFT_max 443.2 s) | ~15% | ~-0.5% .. -0.8% | -0.8% .. -1.2% | +2% .. +6% | | **4x256k** (TTFT_max 609.2 s) | ~12% | ~-0.4% .. -0.6% | -0.6% .. -0.9% | +1.5% .. +4.5% | (Target-shape rows assume 1024 output tokens at 70-90 ms/step. That decode rate is an ESTIMATE; it has not been measured at 128k/256k.) Caveats that must travel with these numbers: - Standalone byte savings have **not** converted 1:1 in-model here. The down_proj bf4 A/B delivered **3.0% against a predicted 15.2%** (`goal3-down-proj-bf4-2026-09-05.jsonl`), and N1 records a 2.35x faster standalone down_proj that ran 0-2% slower in-model. The measured-slope column already discounts for this. The bandwidth column does not. - The sweep arms are goal1 (down bf8). Production's down bf4 is already worth ~2.1-3.1 ms/step *relative to those sweep cells*. Re-baseline on the prod weight mix before crediting any new weight work. - Prefill: bf4 vs bf8 weights made **0% difference at M=512** (`docs/QUANTIZATION.md`, `bench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl`), since prefill is compute-bound. So **no option here touches TTFT**, which dominates at the target shapes. For B, prefill would get *worse*: a reconstruct per chunk plus bf16/HiFi matmuls where today runs LoFi. - With prefix caching now live (commit `4e91d87`), cached heads shrink TTFT, which raises decode's share and makes E1's saving worth somewhat more on repeated-prefix agent workloads. **Memory angle.** The budget is 61.73 GiB usable. Weights take ~25.1 GiB, the target KV is 32 GiB at bf8, trace 0.5-1 GiB, and margin is ~1.9-3.6 GiB (`docs/GOAL-AGENT-SERVING.md`, `prod` profile comment). At bf8 KV, 1 GiB ≈ 32,768 tokens across the pair. | weight mix | Δ vs prod | extra bf8 KV tokens | what it buys (batch is capped at 8) | |---|---|---|---| | E1: attn/GDN→bf4, lm_head bf8 | +3.37 GiB | ~110k | a comfortable margin at 4x256k/8x128k, plus room for prefix-cache blocks and GDN snapshots beyond the live batch | | EXL3 4 bpw (hypothetical B) | +5.16 GiB | ~169k | same, more | | EXL3 3 bpw (hypothetical B) | +7.99 GiB | ~262k | about one extra 256k context's worth. Not usable as a 5th agent (B cap) | Batch is capped at 8 by owner directive, so capacity does not buy "more agents". It buys **margin** (two host freezes on 2026-09-14 came from memory pressure, albeit host-side) and **prefix-cache residency**, which is the enabling feature at the target shapes. ## Integration plan (phases, probes, gates) Principles: every probe **fails closed** and writes a `bench/runs/*.jsonl` record even on failure. Every instrument is exercised on fixtures that force pass, fail and null before it sees real data. Arms **self-identify** from server banners, and the weight arm must log per-group dtype plus the checkpoint/tensor-cache hash, like `patches/tt-metal/0016`. The trap to avoid: `ttnn.as_tensor` reloads a cache file as-is, so a stale `tensor_cache_*` dir silently serves the old weights. A new quantization **must** use a distinct cache suffix. Arms are interleaved on one source tree and differ only by environment or checkpoint path. ### P0: offline error study (desk + off-box GPU, no cards). 3-5 days - Collect per-linear calibration Hessians for Qwen3.8-27B **bf16** on a CUDA box (not cfx-llm2, whose probe budget is ~5.5 GB of host RAM). The exllamav3 converter already does this, and its `sc_trace.py` self-sampled trace matches our agentic/reasoning distribution better than web text. - For each linear group (gate/up, down, attn qkv/o, GDN qkvzab/out, lm_head), compute proxy error `tr(E H E^T)/tr(W H W^T)` for: (i) RTN bfp4_b using an **exact emulation of the tt-metal host packer** (`blockfloat_common.cpp::convert_u32_to_bfp`, per #57410), from both the FP8 and the bf16 source; (ii) RTN bfp8_b; (iii) GPTQ/LDLQ onto the bfp4_b grid; (iv) (iii) + residual rotation; (v) EXL3 4.0/3.0 bpw, reconstructed with `exl3xpu/ref.py` (bit-exact). - **Go** to P2/P3 if (iii) cuts proxy error by >=2x vs (i) on attn/GDN tensors *and* brings it within 2x of (v)-4bpw. **No-go** (stop E1) if (iii) gains <25% over (i) everywhere. ### P1 (E0): accuracy gate, built once and reused. 3-5 days + 2 card holds GSM8K on short prompts is not acceptable here: it is near-blind to KV dtype, and the same problem applies to weights through the long GDN recurrence. Proposed gate, three layers: 1. **Teacher-forced logit gate** (exl3xpu Gate A3 pattern). A sealed panel of N windows. Report KL(ref‖arm), top-1 agreement and max-KL at positions **deep in long contexts**: the last 256 tokens of 4k, 32k and 128k documents. Quantization error in `in_proj_qkvz` compounds through 48 recurrent states, and a 256-token panel cannot see that. The reference is bf16 HF on an off-box GPU. The device side uses vLLM `prompt_logprobs` (probe first that the TT plugin supports it; the host-sampling route that `--logprobs` forces is fine for accuracy, just not for the argmax path). Pre-register thresholds **relative to the current prod arm**: a new weight arm must be <= prod KL at every context. 2. **Task gate at the owner's workload.** Harder keyed retrieval: more needles, dense decoys, multi-hop, at 16k/32k/64k (`bench/kv-accuracy.py`; the 2026-09-13 Gate A sat at its ceiling). Add the 20-turn agentic tool-call canary (`phase2a-prefix-e2e` correctness stage) and a code-generation subset with thinking on. Report the rule-of-three bound, never "equal". 3. **Reproducibility.** A sha self-check at fixed width, at B=4 *and* B=8 (width changes greedy output). A format change legitimately changes outputs, so sha-identity to the old arm is **not** the bar. - First deliverable: **the missing accuracy record for production's bf4 MLP** (RTN from FP8) against bf16 and against goal1 (down bf8). ### P2 (E1a): grid-snapped MLP-only checkpoint. 1-2 weeks + 2-3 card holds - The quantizer writes bf16 safetensors whose values lie exactly on the bfp4_b grid. It simulates the packer's shared-exponent choice, including the case where rounding the block max crosses a binade. Every bfp4_b value is exactly representable in bf16, so the **host packer is the identity** and **no loader or kernel change** is needed. Verify by round trip: host-pack, unpack, compare 100% bitwise on every tensor, offline with ttnn host APIs and no device. - Same bytes as prod, so predicted Δ decode **0 ± 0.5 ms**. A shift beyond that means the arm is contaminated. Gate: P1 KL strictly below prod RTN at every context. Discard if not. - Nix: package the quantizer as `nix run .#bfp-quant` (off-box CUDA, per the one-stop-shop directive) and pin the output checkpoint by hash like any other model input. ### P3 (E1b): attention/GDN projections to bf4. 1-2 weeks + 3-4 card holds - Needs a dtype gate per group (extend `QWEN36_MLP_DOWN_BF4` to a small policy env var, banner-logged). - Pre-registered prediction: **-2.6 to -4.0 ms/step** at B=4 and B=8 (weight read is batch-flat). **Falsifier: < 1.0 ms.** Measured with `bench/ab/sweep-32k.sh` extended with a prod-weight-mix control arm (interleaved), reported as makespan and TTLT, never decode ms alone. - Gate: P1 KL <= prod at every context, and retrieval/agentic within the rule-of-three bound. Go to default only if both hold *and* makespan improves beyond the 0.33% noise floor at >= 3 of 4 contexts. ### P4 (E2/E3, optional). 2-8 weeks - E2 (residual rotation) only if P0 shows (iv) clearly better than (iii) on o/out/down. It needs norm-weight folding checked against Qwen3.5's RMSNorm parameterization, and GDN's in_proj_a/b (bf16) rotated consistently. - E3 (per-tile mixed precision) only if some tensor group fails P3's gate at bf4 but passes at bf8. Prerequisites: #42841 (32-core PCC) and #42586 (hang) fixed upstream, then port `MatmulCustomCompressed` onto our TP=2 DRAM-sharded 1D decode progcfgs. BSPM maps would come from a sensitivity pass (exllamav3 `sc_optimize.py` is the model for this). ### P5 (A, optional, owner's call): "runs EXL3 checkpoints" compatibility. ~1 week - Add `is_exl3_checkpoint` next to the existing `is_fp8_checkpoint` / `is_nvfp4_checkpoint` in `weight_mapping.py`, called from `model_config.load_state_dict`. Reconstruct with a vendored `ref.py` (MIT) to the original basis, then `shard_w` to bf8 (quality-preserving) or bf4. - **Trap:** `ref.weight_orig` builds a full fp32 `k x n`. For lm_head `[248320, 5120]` that is 5.1 GB, which is an instant SIGKILL on cfx-llm2 (same trap as `mtp_head_oracle`). Reconstruct in column chunks. Transcode once off-box into the tensor cache rather than on the serving host. - Gate: reconstructed weights bit-identical to `exllamav3.reconstruct()` (the exl3xpu Gate A1 approach) plus P1. Expect **slower decode** than prod (bf8) or worse attn/GDN quality (bf4). ### P-B (only if someone insists on on-device trellis). Half a day of card time - A single-core SFPU microkernel for the K=4 `mul1` decode (extract, `SFPMUL24` multiply, SWAR byte-sum, cast, FMA, store) over L1-resident words. It reads the cycle counter and records cycles per 32-weight row. - **Go only if <= 2.0 cycles/row** (serial break-even). Between 2 and 5.6 would need proven decode/DRAM overlap *and* FPU concurrency, which is a second probe. Above 5.6: close permanently, in writing. Expected (ESTIMATE): 10-20 cycles/row, so close. Effort total for the recommended path (P0-P3): **~5-8 engineer-weeks and ~8-10 card holds**, every one serialized on `/tmp/ttlock` with the `MemAvailable` >= 4 GB check. ## Risks / unknowns - **Conversion of bytes to ms is weak here** (3.0% delivered vs 15.2% predicted). E1b's -2.6 to -4.0 ms could land near zero. The falsifier is set at 1.0 ms. - **bf4 attention/GDN may not pass any reasonable gate.** unsloth's NVFP4 recipe keeps attn/linear_attn/lm_head and the last 8 MLP layers at 8-bit; QUASAR gets all-4-bit only via QAT (`docs/QUANTIZATION.md`). GPTQ-style rounding narrows but may not close that gap. - **The recurrent state compounds error.** 48 GDN layers carry state across up to 256k tokens. Short-panel metrics can pass while long contexts drift. That is why P1 measures deep positions. - **The bfp exponent axis is along N** (`[K,N]` tile layout). Rotations help only when applied along N. Verify the actual on-device layout of each weight (some matmul configs may transpose) before designing E2. - **Host packer semantics may change upstream** (#57410 proposes aligning the host to the device). The grid-snapped checkpoint stays safe only if values are exactly representable, so the packer does not round at all. The round-trip test in P2 must re-run on every tt-metal pin bump. - **Upstream compressed-matmul bugs** (#42841, #42586) block E3 on exactly our core counts (32/33/44). - **Off-box compute** is needed for Hessians and quantization (27B, ~2 h per full conversion on one fast GPU per exllamav3 `doc/optimize.md`, THIRD-PARTY). cfx-llm2 cannot host it. - **Side observation, not a proposal:** exl3xpu runs vLLM's batched MTP (`qwen3_5_mtp`, k=3) on this exact model with acceptance ~2.4-3.1 at C=1..16 and 2-2.4x aggregate at C=8 (THIRD-PARTY). Owner directive `0aaaaa` dropped MTP, and "do not propose speculation" stands. This is recorded only because the directive says MTP was never measured here and must never be recorded as rejected on evidence. ## Sources **This repo (MEASURED / internal):** - `bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl`: per-projection read/non-read split, 454-469 GB/s, bytes/element 0.5625/1.0625 - `bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonl`: per-group element counts (basis of all byte arithmetic) - `bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl`: down_proj bf4 in-model: +3.1% vs +15.2% predicted, accuracy unverified - `bench/runs/sweep-32k-20260914T142154Z.jsonl` and `.table.txt`: makespan/TTFT/decode, goal1 profile (no down bf4) - `bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl`, `...B4-256k-20260914T134045Z.jsonl` (CLAUDE.md cites a non-existent `...134735Z`): 443.2 s / 609.2 s - `bench/runs/gateA-kv-accuracy-20260913T215707Z.jsonl`: retrieval gate design and ceiling problem - `bench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl`: bf4 vs bf8 at LoFi (0% at prefill M=512; PCC 0.9932 vs 0.9999) - `scripts/vllm-tt-serve.sh:155` (prod sets `QWEN36_MLP_DOWN_BF4=1`); `bench/ab/sweep-32k.sh:57,82` (arms lack it) - `patches/prod-local-metal/models/demos/blackhole/qwen36/tt/{model_config.py,tp_common.py,model.py}`: loader (`load_state_dict`, `is_fp8/nvfp4_checkpoint`), `shard_w` `[in,out]` layout, lm_head bf8 - `docs/QUANTIZATION.md`, `docs/VRAM-BUDGET.md`, `docs/SATURATION-AND-SRAM.md`, `docs/GOAL-AGENT-SERVING.md`, `docs/PERFORMANCE.md` **THIRD-PARTY:** - exl3xpu: https://github.com/0xSero/exl3xpu @ `2d17c57`: `README.md`, `docs/DESIGN.md`, `docs/PROGRESS.md`, `docs/GOAL.md`, `csrc/exl3_esimd.h`, `exl3xpu/ref.py`, `exl3xpu/ops.py`, `models/qwen3.8-27b-exl3-4.00bpw/recipe.json`, `tests/gateA3/` - exllamav3: https://github.com/turboderp-org/exllamav3 @ `6b84a21`: `doc/convert.md`, `doc/optimize.md`, `doc/llama31_8b_instruct_kld_bpw.png`, `exllamav3/modules/quant/exl3_lib/quantize.py` - QTIP: Tseng et al., arXiv:2406.11235 (NeurIPS 2024) - Checkpoint: https://huggingface.co/turboderp/Qwen3.8-27B-exl3 (branch `4.00bpw`, rev `113cf7a`) - Blackhole SFPU `SFPMUL24`: https://github.com/tenstorrent/tt-isa-documentation/blob/main/BlackholeA0/TensixTile/TensixCoprocessor/SFPMUL24.md - 32-bit integer multiply on Blackhole, 8 cycles/row: https://www.jasondavies.com/2025/tenstorrent-multiply-int32/ - tt-metal #57410 (host vs device bfp4 rounding), #38520 / #39989 / #42601 / #42586 / #42841 (compressed per-tile matmul, BSPM), #37857 (Blitz DeepSeek weight-format spec) - Searches with zero hits: tt-metal issues/PRs and tenstorrent org PRs for "trellis", "QTIP", "exllama", "exl3", "codebook", "lookup table dequant" (2026-09-24)
Author
Owner

The EXL3 trellis-on-SFPU no-go rests on a desk estimate (25-50 ms decode). Please keep this open with an ESTIMATE label until a one-kernel SFPU trellis-decode microbench is measured on Blackhole.

Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).

The EXL3 trellis-on-SFPU no-go rests on a desk **estimate** (25-50 ms decode). Please keep this open with an ESTIMATE label until a one-kernel SFPU trellis-decode microbench is measured on Blackhole. Reopened under the new rule (owner, 2026-09-25): **a closure needs an on-hardware record at the stated scope**; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).
Sign in to join this conversation.
No labels
human-approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bitpartner/tt-stack#61
No description provided.