research: exl3xpu / EXL3 trellis quant on Blackhole — port is no-go; EXL3-style offline bf4 quantizer + accuracy gate instead #61
Labels
No labels
human-approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack#61
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
TL;DR verdict
Porting exl3xpu as it stands is a NO-GO. It would make decode slower, not faster. There are two
cheaper things worth doing instead, and both use EXL3's ideas while leaving its bitstream behind.
vLLM plugin. Its kernels are SYCL/ESIMD built around Xe2
dpas(VNNI) anddp4a. It borrows GDNand attention from vLLM-XPU and has no tensor parallelism. Nothing in it runs on Tensix. What
carries over is the format spec, the bit-exact PyTorch reference decoder (
exl3xpu/ref.py, MIT),and the accuracy-gate method.
MEASURED). Per card, each decode step would have to turn about 12.2 B trellis weights into tiles.
On Blackhole that work lands on the SFPU (32 lanes per Tensix;
SFPMUL24is the only integermultiply, and it is 23-bit). Every weight needs a 16-bit window extract, a 16x32-bit multiply
mod 2^32, a byte-sum and one fp FMA, which is roughly 15-20 SFPU ops per 32-weight row. That puts
the decode at ~25-50 ms per step. The most 4-bpw EXL3 could save against production's weight
read is ~4-6 ms per step, and only if the decode were free. Break-even needs <=2 SFPU cycles
per 32-weight row if decode runs serially, or <=5.6 if it overlaps the DRAM read perfectly. Doing
the dequant in the RISC-V reader kernels is about 50x worse (~0.7 s per token).
37% of the decode step.-1% atThe bytes EXL3 would save relative to production (which already runs all MLP at bf4) are worth
-4 to -6 ms per step. That is -4% to -10% makespan on the <=32k sweep and **
8x128k / 4x256k**, where serialised prefill (443 s / 609 s) dominates. For comparison, prefix
caching and prefill work each move those shapes by tens of percent.
mlp.down_projat bf4 (
QWEN36_MLP_DOWN_BF4=1in theprodprofile). Its only record says "accuracy:unverified against a bf8 baseline" (
bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl). Thiscosts nothing at runtime and is needed before any weight-format change.
bfp4_b. It rounds withHessian awareness (GPTQ/LDLQ, EXL3's error-feedback step) straight onto the bfp4_b grid and
emits a grid-snapped bf16 checkpoint. The host packer reproduces those bits exactly, so there
is zero runtime change. Value: better bf4 quality where we already use it, and possibly
enough quality to move attention/GDN projections from bf8 to bf4. That second step is worth
about -2.6 to -4.0 ms per step and +3.4 GiB of card memory (~110k bf8 KV tokens) (ESTIMATE).
Tenstorrent's own
MatmulCustomCompressedplus BitSculpt "BSPM" precision maps. It is thenative equivalent of EXL3's per-tensor bitrate recipes. It exists in tt-metal (
deepseek_v3_b1)but has open correctness bugs at 32 cores.
but it is a compatibility feature, not a performance one. Transcoding EXL3 to bf8 keeps EXL3's
quality but reads more bytes than production. Transcoding to bf4 quantizes twice and loses to our
current source on attention/GDN.
Correction to the brief: production is not "gate/up bf4, down bf8". The
prodprofile setsQWEN36_MLP_DOWN_BF4=1(scripts/vllm-tt-serve.sh:155), so production streams ~9.33 GB/card/token.The 10.76 GB/card figure belongs to the
goal1profile, which is what the benchmark drivers use(
bench/ab/sweep-32k.sh,decode-serving-ab.sh). Consequence: the 2026-09-14 "clean-slate"sweep's optimized arm is not production's weight mix. Its record's
env_basehas noQWEN36_MLP_DOWN_BF4.What exl3xpu is
Repo:
github.com/0xSero/exl3xpu@2d17c57(2026-09-24), MIT. It is a vLLM 0.26.1 XPUplugin (
vllm.general_pluginsentry point,register_quantization_config("exl3")).Hardware: Intel Arc Pro B70 (BMG-G31, Xe2, 32 GB). "xpu" is PyTorch/oneAPI's Intel GPU
device, not a generic accelerator. Kernels are SYCL ESIMD (
csrc/exl3_esimd.h, 767 lines),JIT-compiled with icpx (
spir64).It consumes turboderp's EXL3 checkpoints (exllamav3 v1.x format). The served model is
exactly ours:
turboderp/Qwen3.8-27B-exl3@ 4.00 bpw (lm_head 6 bpw,mul1codebook, 401EXL3 tensors, 14.91 GiB).
in_proj_a/b, norms, embeddings and the vision tower stay bf16.Scope: vLLM supplies scheduling, paged KV, and the GDN and attention kernels (the XPU ones).
exl3xpu only replaces the linear layers. No tensor parallelism ("use data parallel").
Only 4 and 6 bpw
mul1are enabled by default; 2/3/5 bpw and the mcg/3INST codebooks need-DEXL3_ALL_CODEBOOKS.Correctness: dequantized weights are bit-identical to exllamav3
reconstruct()for all 401tensors on every kernel path. Logits vs exllamav3 on an RTX 3090: top-1 99.63%, KL 9.8e-5
on a sealed 64x256 teacher-forced panel ("Gate A3"). This is implementation equivalence
against EXL3 itself, not quantization quality against bf16.
Performance, THIRD-PARTY (one B70,
bench/results/2026-09-23.jsonl,recipe.json):docs/GOAL.mdT4 floor)The harnesses and definitions differ, so this is not an A/B. Our records:
bench/runs/serving-16k-20260913T095210Z.jsonl,bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl,bench/runs/target-shape-ttft-B4-256k-20260914T134045Z.jsonl,bench/runs/sweep-32k-20260914T142154Z.jsonl.Its own diagnosis is relevant here: on Xe2 the M=1 GEMV is co-limited by integer-ALU issue
(~22-25 ms) and DRAM (~23.5 ms). The trellis decode sits "near the integer-ALU roofline", on
a GPU with far more integer throughput per weight than a Tensix SFPU (see below).
EXL3 format & kernel anatomy
EXL3 is turboderp's implementation of QTIP (Tseng et al., NeurIPS 2024: Quantization with
Trellises and Incoherence Processing). The quantizer has three components, and only one of them
is tied to the bitstream:
W = diag(suh) · H · W_inner · H · diag(svh), whereHis thenormalized 128x128 Sylvester Hadamard applied blockwise and
suh/svhare fp16 sign/scalevectors. At inference:
y = had128(had128(x * suh) @ W_inner) * svh. This means an inputHadamard and an output Hadamard per linear.
H = X^T X, which isregularized and LDL-decomposed. Tiles are quantized in order with error feedback
(
exllamav3/modules/quant/exl3_lib/quantize.py:549 ldlq,:1471 quantize_exl3). Theconverter's proxy error is
tr(E H E^T) / tr(W H W^T).256*Kbits (K =bpw, 1..8). Value
tis the 16-bit window ending at bit(t+1)*K, circular within thetile, placed at a tensor-core permutation. The
mul1codebook, bit-exact perref.py:x = state * 0x83DCD12D (mod 2^32), thenv = fp16(1024 + bytesum(x)) * fp16(0x1EEE) + fp16(0xC931), one fp16 FMA with a single rounding.Per weight, the inner loop is: funnel-shift two 32-bit words, mask to 16 bits, 32-bit integer
multiply, sum 4 bytes, int to fp16, one FMA. The Xe2 kernel does this in about 9 SIMD32
instructions per 32 values:
shl/shr/or/mul/dp4a/mov/mad.dp4a(0x6400, x, 0x01010101)doesthe byte-sum and fp16 bias in one instruction, and that trick is the reason it is fast. Memory
access is a pure contiguous stream of trellis words per tile row (K=4 packs 8 values per word).
The 128-point Hadamards are separate small kernels (~1.2 ms per step under graphs). For M>128
(prefill) the plugin reconstructs fp16
W_innerslices and calls oneDNN GEMM.Available bitrates: 1-8 bpw per tensor, with per-tensor recipes from
sc_optimize.py. THIRD-PARTYquality, exllamav3
doc/llama31_8b_instruct_kld_bpw.png(Llama-3.1-8B, KL vs bf16): EXL3 3.0bpw 0.053, 3.5 bpw 0.030, 4.0 bpw 0.013, 5.0 bpw 0.003; GGUF Q4_K_M 0.018, Q6_K 0.003. Nobody has
measured KL for RTN
bfp4_bon our model. That is the gap E0/E1 close.Qwen3.x hybrid/GDN support: yes. exllamav3 quantizes the GDN projections (
in_proj_qkvz,out_proj) like any linear, and exl3xpu serves the 48-GDN/16-attention model through vLLM's GDNkernels. Nothing in the format is GDN-specific.
Blackhole mapping
Hardware facts (MEASURED or from source,
docs/SATURATION-AND-SRAM.md): p150a is 2x-harvested,110 compute Tensix (a column is reserved for dispatch), 1.5 MiB L1 each (165 MiB per card),
AICLK 1350 MHz, 8 GDDR6 channels. Usable DRAM is 30.87 GiB per card (
docs/VRAM-BUDGET.md).Decode matmul readers reach 454-469 GB/s (
bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl).What the Tensix pipeline gives us for free: the unpacker decompresses
bfp8_b/bfp4_b/bfp2_bin hardware. Block-float is one shared 8-bit exponent per 16 values (a face row), with7/3/1-bit magnitude plus sign, which is 1.0625 / 0.5625 / 0.3125 B per element. Those are the only
sub-8-bit formats Blackhole can stream at line rate. There is no codebook or LUT unpack mode, and no
MXFP4/NVFP4 (those are Quasar-only;
docs/QUANTIZATION.md). A weight tile laid out[K, N](
tp_common.shard_wtransposes to[in, out]) shares exponents along N, across 16 outputchannels at the same input index.
Where a trellis decode could run
SFPMUL24(23x23-bit, Blackhole-only)SFPMUL24+ adds/shifts. Byte-sum via SWAR is about 6 ops. Extract about 4 ops, convert+FMA about 2, plus load/store. ~15-20 ops per 32-weight row, about 10-20 cycles withSFPLOADMACROoverlap. Time = 2.56 ms x cycles/row, so ~25-50 ms per stepReference points for the SFPU estimate (THIRD-PARTY): a full 32x32-bit integer multiply on Blackhole
is 8 cycles per 32-value row even hand-scheduled with
SFPLOADMACRO(jasondavies.com, 2025-11).Our multiply is cheaper (16-bit operand), but the byte-sum has no
dp4aequivalent. Two structuralcosts come on top. First, decoded values leave through the packer to L1 and come back through the
unpacker into SrcB for the FPU, which is an extra round trip per tile. Second, SFPU and FPU are
driven by the same math thread, so decode and matmul largely serialize within a core.
Break-even (ESTIMATE): production reads 9.33 GB/card/token, which is 20.6 ms at 454 GB/s. 4-bpw EXL3
reads 6.57 GB/card, 14.5 ms. With a serial decode, EXL3 wins only if decode takes under ~4-6 ms,
which means <=1.6-2.4 cycles per 32-weight row. With decode perfectly overlapped against DRAM,
it wins only if decode finishes under 14.5 ms, which means <=5.6 cycles per row, and even then
the gain caps at ~6 ms. Neither is credible given an 8-cycle hand-tuned 32-bit multiply alone.
Hadamards on Tensix
An online
had128is a[M*k/128, 128] @ H128matmul, cheap in FLOPs. But EXL3 needs one in andone out per linear, plus one copy per fused constituent because each has its own
suh. That is~400-640 extra traced ops per decode step. At ~5-15 µs device time each, that is ~2-9 ms
(ESTIMATE). It has to be fused into the matmul reader/epilogue or folded offline to be free. For
TP=2 the blocks divide cleanly: row-parallel K shards are 8704 = 68x128 and 3072 = 24x128, and
because the Hadamard is linear, the output one can be applied before or after the all-reduce. Only a
residual-stream rotation (QuaRot-style) can be folded into weights at zero runtime cost. It makes
the output (N) dim of
o_proj/out_proj/down_projincoherent, which is the axis Tensix bfpexponents are shared along. For
qkv/in_proj/gate_upit rotates K, which is the wrong axis forbfp blocks (hypothesis, testable offline).
Prior art on Tenstorrent
and tenstorrent org PRs for trellis, QTIP, exllama, exl3, codebook and "lookup table dequant";
zero hits.
MatmulCustomCompressedplusCompressedTensorinmodels/demos/deepseek_v3_b1/, with per-32x32-tile format codes {bfp8, bfp4, bfp2, zero} driven byBitSculpt "BSPM" precision maps (
compressed_tensor/bspm_loader.py; issues #38520, #39989).THIRD-PARTY: DeepSeek experts at 3.5 bits/element (78% bfp4, 22% zero) passed Lite PPL 2.98 and
AIME24 85.4%, and failed Full PPL by +0.08 (#42601). Open defects: wrong PCC (~0.04) at 32 cores
with any non-bfp8 tile (#42841) and a hang with zero bfp8 tiles (#42586).
ttnn.from_torch(..., bfloat4_b)(the host packer, whichis how our bf4 weights are made) rounds straight to 3 mantissa bits (RNE). The device
typecastrounds to BFP8 first and then truncates. They differ on 42.5% of elements. The unpack side is
correct. Two consequences for us. First, the host is free to write any bits we choose, which
is what makes E1's "grid-snapped checkpoint" work. Second, any on-device re-quantization (e.g. of
KV or activations) does not match host-made weights bit for bit.
Options compared
"Δ decode" is against production (all-MLP bf4, 9.33 GB/card/token). The two numbers are the
measured in-model slope (0.78 ms/GiB saved, from
goal3-down-proj-bf4-2026-09-05.jsonl, takenon an older stack) and the bandwidth-ideal figure (454 GB/s). Both are ESTIMATES.
had128in/outW_inneras bfp + online hadByte model (from
bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonlelement counts, streamedper token, embed excluded): goal1 21.52 GB total / 10.76 per card; prod 18.67 / 9.33; all-linear
bf4 with lm_head bf8 15.05 / 7.52; EXL3 4 bpw (head 6) 13.13 / 6.57; EXL3 3 bpw 10.09 / 5.04.
Does it pay here? (with numbers)
Makespan model. Each cell of the 2026-09-14 sweep satisfies makespan ≈ TTFT_max + 1024 ×
decode_ms to within 1.3 s (MEASURED inputs,
sweep-32k-20260914T142154Z.table.txt; e.g. B=8/32k:80.84 + 67.80 = 148.6 vs 148.72 measured). A per-step saving Δ therefore moves makespan by
1.024·Δ seconds. Decode's share of makespan is 88% (B=4/4k) down to 46% (B=8/32k). (This
contradicts the "prefill is 74-79% of wall" wording in CLAUDE.md
0aaaaaaaaaaaaaaaaa: by thisidentity it is decode that is ~72-78% of wall at B=8/4k-8k.)
(Target-shape rows assume 1024 output tokens at 70-90 ms/step. That decode rate is an ESTIMATE; it
has not been measured at 128k/256k.)
Caveats that must travel with these numbers:
delivered 3.0% against a predicted 15.2% (
goal3-down-proj-bf4-2026-09-05.jsonl), and N1records a 2.35x faster standalone down_proj that ran 0-2% slower in-model. The measured-slope
column already discounts for this. The bandwidth column does not.
relative to those sweep cells. Re-baseline on the prod weight mix before crediting any new
weight work.
docs/QUANTIZATION.md,bench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl), since prefill is compute-bound. Sono option here touches TTFT, which dominates at the target shapes. For B, prefill would get
worse: a reconstruct per chunk plus bf16/HiFi matmuls where today runs LoFi.
4e91d87), cached heads shrink TTFT, which raises decode'sshare and makes E1's saving worth somewhat more on repeated-prefix agent workloads.
Memory angle. The budget is 61.73 GiB usable. Weights take ~25.1 GiB, the target KV is 32 GiB at
bf8, trace 0.5-1 GiB, and margin is ~1.9-3.6 GiB (
docs/GOAL-AGENT-SERVING.md,prodprofilecomment). At bf8 KV, 1 GiB ≈ 32,768 tokens across the pair.
Batch is capped at 8 by owner directive, so capacity does not buy "more agents". It buys margin
(two host freezes on 2026-09-14 came from memory pressure, albeit host-side) and prefix-cache
residency, which is the enabling feature at the target shapes.
Integration plan (phases, probes, gates)
Principles: every probe fails closed and writes a
bench/runs/*.jsonlrecord even on failure.Every instrument is exercised on fixtures that force pass, fail and null before it sees real data.
Arms self-identify from server banners, and the weight arm must log per-group dtype plus the
checkpoint/tensor-cache hash, like
patches/tt-metal/0016. The trap to avoid:ttnn.as_tensorreloads a cache file as-is, so a stale
tensor_cache_*dir silently serves the old weights. A newquantization must use a distinct cache suffix. Arms are interleaved on one source tree and
differ only by environment or checkpoint path.
P0: offline error study (desk + off-box GPU, no cards). 3-5 days
whose probe budget is ~5.5 GB of host RAM). The exllamav3 converter already does this, and its
sc_trace.pyself-sampled trace matches our agentic/reasoning distribution better than web text.tr(E H E^T)/tr(W H W^T)for:(i) RTN bfp4_b using an exact emulation of the tt-metal host packer
(
blockfloat_common.cpp::convert_u32_to_bfp, per #57410), from both the FP8 and the bf16 source;(ii) RTN bfp8_b;
(iii) GPTQ/LDLQ onto the bfp4_b grid;
(iv) (iii) + residual rotation;
(v) EXL3 4.0/3.0 bpw, reconstructed with
exl3xpu/ref.py(bit-exact).2x of (v)-4bpw. No-go (stop E1) if (iii) gains <25% over (i) everywhere.
P1 (E0): accuracy gate, built once and reused. 3-5 days + 2 card holds
GSM8K on short prompts is not acceptable here: it is near-blind to KV dtype, and the same problem
applies to weights through the long GDN recurrence. Proposed gate, three layers:
KL(ref‖arm), top-1 agreement and max-KL at positions deep in long contexts: the last 256
tokens of 4k, 32k and 128k documents. Quantization error in
in_proj_qkvzcompounds through 48recurrent states, and a 256-token panel cannot see that. The reference is bf16 HF on an off-box
GPU. The device side uses vLLM
prompt_logprobs(probe first that the TT plugin supports it; thehost-sampling route that
--logprobsforces is fine for accuracy, just not for the argmax path).Pre-register thresholds relative to the current prod arm: a new weight arm must be <= prod
KL at every context.
multi-hop, at 16k/32k/64k (
bench/kv-accuracy.py; the 2026-09-13 Gate A sat at its ceiling).Add the 20-turn agentic tool-call canary (
phase2a-prefix-e2ecorrectness stage) and acode-generation subset with thinking on. Report the rule-of-three bound, never "equal".
output). A format change legitimately changes outputs, so sha-identity to the old arm is not
the bar.
against bf16 and against goal1 (down bf8).
P2 (E1a): grid-snapped MLP-only checkpoint. 1-2 weeks + 2-3 card holds
the packer's shared-exponent choice, including the case where rounding the block max crosses a
binade. Every bfp4_b value is exactly representable in bf16, so the host packer is the
identity and no loader or kernel change is needed. Verify by round trip: host-pack, unpack,
compare 100% bitwise on every tensor, offline with ttnn host APIs and no device.
contaminated. Gate: P1 KL strictly below prod RTN at every context. Discard if not.
nix run .#bfp-quant(off-box CUDA, per the one-stop-shopdirective) and pin the output checkpoint by hash like any other model input.
P3 (E1b): attention/GDN projections to bf4. 1-2 weeks + 3-4 card holds
QWEN36_MLP_DOWN_BF4to a small policy env var, banner-logged).Falsifier: < 1.0 ms. Measured with
bench/ab/sweep-32k.shextended with a prod-weight-mixcontrol arm (interleaved), reported as makespan and TTLT, never decode ms alone.
Go to default only if both hold and makespan improves beyond the 0.33% noise floor at >= 3 of
4 contexts.
P4 (E2/E3, optional). 2-8 weeks
norm-weight folding checked against Qwen3.5's RMSNorm parameterization, and GDN's in_proj_a/b
(bf16) rotated consistently.
bf8. Prerequisites: #42841 (32-core PCC) and #42586 (hang) fixed upstream, then port
MatmulCustomCompressedonto our TP=2 DRAM-sharded 1D decode progcfgs. BSPM maps would comefrom a sensitivity pass (exllamav3
sc_optimize.pyis the model for this).P5 (A, optional, owner's call): "runs EXL3 checkpoints" compatibility. ~1 week
is_exl3_checkpointnext to the existingis_fp8_checkpoint/is_nvfp4_checkpointinweight_mapping.py, called frommodel_config.load_state_dict. Reconstruct with a vendoredref.py(MIT) to the original basis, thenshard_wto bf8 (quality-preserving) or bf4.ref.weight_origbuilds a full fp32k x n. For lm_head[248320, 5120]that is5.1 GB, which is an instant SIGKILL on cfx-llm2 (same trap as
mtp_head_oracle). Reconstruct incolumn chunks. Transcode once off-box into the tensor cache rather than on the serving host.
exllamav3.reconstruct()(the exl3xpu Gate A1approach) plus P1. Expect slower decode than prod (bf8) or worse attn/GDN quality (bf4).
P-B (only if someone insists on on-device trellis). Half a day of card time
mul1decode (extract,SFPMUL24multiply, SWARbyte-sum, cast, FMA, store) over L1-resident words. It reads the cycle counter and records
cycles per 32-weight row.
decode/DRAM overlap and FPU concurrency, which is a second probe. Above 5.6: close permanently,
in writing. Expected (ESTIMATE): 10-20 cycles/row, so close.
Effort total for the recommended path (P0-P3): ~5-8 engineer-weeks and ~8-10 card holds,
every one serialized on
/tmp/ttlockwith theMemAvailable>= 4 GB check.Risks / unknowns
-4.0 ms could land near zero. The falsifier is set at 1.0 ms.
attn/linear_attn/lm_head and the last 8 MLP layers at 8-bit; QUASAR gets all-4-bit only via QAT
(
docs/QUANTIZATION.md). GPTQ-style rounding narrows but may not close that gap.Short-panel metrics can pass while long contexts drift. That is why P1 measures deep positions.
[K,N]tile layout). Rotations help only when applied alongN. Verify the actual on-device layout of each weight (some matmul configs may transpose) before
designing E2.
The grid-snapped checkpoint stays safe only if values are exactly representable, so the packer
does not round at all. The round-trip test in P2 must re-run on every tt-metal pin bump.
fast GPU per exllamav3
doc/optimize.md, THIRD-PARTY). cfx-llm2 cannot host it.qwen3_5_mtp, k=3) on thisexact model with acceptance ~2.4-3.1 at C=1..16 and 2-2.4x aggregate at C=8 (THIRD-PARTY). Owner
directive
0aaaaadropped MTP, and "do not propose speculation" stands. This is recorded onlybecause the directive says MTP was never measured here and must never be recorded as rejected
on evidence.
Sources
This repo (MEASURED / internal):
bench/runs/n1-matmul-read-compute-split-20260912T174500Z.jsonl: per-projection read/non-read split, 454-469 GB/s, bytes/element 0.5625/1.0625bench/runs/gate0a-weight-traffic-20260911T154154Z.jsonl: per-group element counts (basis of all byte arithmetic)bench/runs/goal3-down-proj-bf4-2026-09-05.jsonl: down_proj bf4 in-model: +3.1% vs +15.2% predicted, accuracy unverifiedbench/runs/sweep-32k-20260914T142154Z.jsonland.table.txt: makespan/TTFT/decode, goal1 profile (no down bf4)bench/runs/target-shape-ttft-B8-128k-20260914T130939Z.jsonl,...B4-256k-20260914T134045Z.jsonl(CLAUDE.md cites a non-existent...134735Z): 443.2 s / 609.2 sbench/runs/gateA-kv-accuracy-20260913T215707Z.jsonl: retrieval gate design and ceiling problembench/runs/goal4-fidelity-dtype-probe-2026-09-05.jsonl: bf4 vs bf8 at LoFi (0% at prefill M=512; PCC 0.9932 vs 0.9999)scripts/vllm-tt-serve.sh:155(prod setsQWEN36_MLP_DOWN_BF4=1);bench/ab/sweep-32k.sh:57,82(arms lack it)patches/prod-local-metal/models/demos/blackhole/qwen36/tt/{model_config.py,tp_common.py,model.py}: loader (load_state_dict,is_fp8/nvfp4_checkpoint),shard_w[in,out]layout, lm_head bf8docs/QUANTIZATION.md,docs/VRAM-BUDGET.md,docs/SATURATION-AND-SRAM.md,docs/GOAL-AGENT-SERVING.md,docs/PERFORMANCE.mdTHIRD-PARTY:
2d17c57:README.md,docs/DESIGN.md,docs/PROGRESS.md,docs/GOAL.md,csrc/exl3_esimd.h,exl3xpu/ref.py,exl3xpu/ops.py,models/qwen3.8-27b-exl3-4.00bpw/recipe.json,tests/gateA3/6b84a21:doc/convert.md,doc/optimize.md,doc/llama31_8b_instruct_kld_bpw.png,exllamav3/modules/quant/exl3_lib/quantize.py4.00bpw, rev113cf7a)SFPMUL24: https://github.com/tenstorrent/tt-isa-documentation/blob/main/BlackholeA0/TensixTile/TensixCoprocessor/SFPMUL24.mdThe EXL3 trellis-on-SFPU no-go rests on a desk estimate (25-50 ms decode). Please keep this open with an ESTIMATE label until a one-kernel SFPU trellis-decode microbench is measured on Blackhole.
Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).