bitpartner/tt-stack

Watch

Fork 0

Quantize on TT cards: CUDA-free GPTQ/LDLQ bfp4 pipeline (Hessians, GPTQ sweep, FP8 teacher on Blackhole) #67

New issue

Open

opened 2026-09-25 08:39:50 +02:00 by Grok · 3 comments

Grok commented

2026-09-25 08:39:50 +02:00

Owner

Owner directive (2026-09-25): the bfp4 GPTQ quantization (#62) must be runnable on our own TT cards, as part of the toolkit, with no CUDA dependency. The first pass used a CUDA box (cfx-llm0, RTX 3080) for the Hessian collection and GPTQ/LDLQ, and a CUDA vLLM for the FP8 teacher.

What the pipeline needs, and where each part would run on TT

Step	Compute shape	TT plan
1. Calibration forward + Hessian `H = Σ XᵀX` per linear input	Model forward plus large rank-k accumulations: pure matmul	Device. Reuse our Qwen36 TT forward and tap each layer's linear inputs, accumulating `XᵀX` in fp32 on device. The hook points are the same ones the DSpark tap capture uses.
2. Cholesky / inverse of H (5120², 17408², 6144²…)	Dense factorization, once per group	Host CPU at first (numpy/LAPACK, fp32/fp64; a 17408² fp64 matrix is 2.4 GB, so stream it on hosts with little RAM). Blocked Cholesky on device later, if it matters.
3. GPTQ/LDLQ column sweep	Serial rounding inside 128-column blocks, then a matmul error propagation `W[:, rest] -= E·Hinv[blk, rest]`	Propagation matmul on device, which is ~all the FLOPs. The in-block serial rounding with the bit-exact bfp4_b rounder (`tools/bfp-quant/bfp.py`) runs on host or as a small device op.
4. Packing to bfp4-packed-v1	Elementwise	Host, already written (`bfp4_unpack.py` / packer).
5. Quality gate vs FP8 (KL / top-1)	Teacher forward with logprobs	Device. The FP8 prod stack is the teacher on TT. This removes the CUDA vLLM teacher windows and their downtime on cfx-llm0.

Risks to measure (on hardware, not ttsim)

Precision. GPTQ is sensitive to H conditioning. TT matmul needs fp32 inputs with fp32_dest_acc_en at HiFi4 (never LoFi together with fp32 acc; a trap we've already paid for). Gate: the TT-built checkpoint must match the CUDA-built G2 in the proxy error tr(E H Eᵀ)/tr(W H Wᵀ) vs W8 to within 1%. Ideally the rounding is also identical on ≥99% of blocks.
Host RAM. cfx-llm2 has ~5.5 GB usable per tt-metal process, so stream per layer. H matrices go to disk.
Card time. Share cards via the tt-card-slot lock (one card per docker container, if the MTP work proves that out).

Deliverables

tools/tt-quant/: collect_hessians_tt.py, gptq_tt.py and teacher_tt.py, plus nix run .#tt-quant.
Records in bench/runs comparing the TT-built and CUDA-built G2: proxy error, block agreement, wall time per stage.
A repo skill, .claude/skills/tt-quant/SKILL.md.

Owner directive (2026-09-25): the bfp4 GPTQ quantization (#62) must be runnable **on our own TT cards**, as part of the toolkit, with no CUDA dependency. The first pass used a CUDA box (cfx-llm0, RTX 3080) for the Hessian collection and GPTQ/LDLQ, and a CUDA vLLM for the FP8 teacher. ## What the pipeline needs, and where each part would run on TT | Step | Compute shape | TT plan | |---|---|---| | 1. Calibration forward + Hessian `H = Σ XᵀX` per linear input | Model forward plus large rank-k accumulations: pure matmul | Device. Reuse our Qwen36 TT forward and tap each layer's linear inputs, accumulating `XᵀX` in fp32 on device. The hook points are the same ones the DSpark tap capture uses. | | 2. Cholesky / inverse of H (5120², 17408², 6144²…) | Dense factorization, once per group | Host CPU at first (numpy/LAPACK, fp32/fp64; a 17408² fp64 matrix is 2.4 GB, so stream it on hosts with little RAM). Blocked Cholesky on device later, if it matters. | | 3. GPTQ/LDLQ column sweep | Serial rounding inside 128-column blocks, then a matmul error propagation `W[:, rest] -= E·Hinv[blk, rest]` | Propagation matmul on device, which is ~all the FLOPs. The in-block serial rounding with the bit-exact bfp4_b rounder (`tools/bfp-quant/bfp.py`) runs on host or as a small device op. | | 4. Packing to bfp4-packed-v1 | Elementwise | Host, already written (`bfp4_unpack.py` / packer). | | 5. Quality gate vs FP8 (KL / top-1) | Teacher forward with logprobs | Device. The FP8 prod stack is the teacher on TT. This removes the CUDA vLLM teacher windows and their downtime on cfx-llm0. | ## Risks to measure (on hardware, not ttsim) - **Precision.** GPTQ is sensitive to H conditioning. TT matmul needs fp32 inputs with `fp32_dest_acc_en` at HiFi4 (never LoFi together with fp32 acc; a trap we've already paid for). Gate: the TT-built checkpoint must match the CUDA-built G2 in the proxy error `tr(E H Eᵀ)/tr(W H Wᵀ)` vs W8 to within 1%. Ideally the rounding is also identical on ≥99% of blocks. - **Host RAM.** cfx-llm2 has ~5.5 GB usable per tt-metal process, so stream per layer. H matrices go to disk. - **Card time.** Share cards via the tt-card-slot lock (one card per docker container, if the MTP work proves that out). ## Deliverables - `tools/tt-quant/`: collect_hessians_tt.py, gptq_tt.py and teacher_tt.py, plus `nix run .#tt-quant`. - Records in bench/runs comparing the TT-built and CUDA-built G2: proxy error, block agreement, wall time per stage. - A repo skill, `.claude/skills/tt-quant/SKILL.md`.

Grok referenced this issue from a commit

2026-09-25 08:52:04 +02:00

tt-quant: CUDA-free GPTQ scaffolding -- device block propagation behind the bfp-quant algorithm

Grok commented

2026-09-25 09:20:25 +02:00

Author

Owner

Progress (branch tt-quant, cdf66ac): tooling for all 5 stages is written and tested on host; card work is queued behind the MTP lock on cfx-llm2.

tools/tt-quant/ttq.py: GPTQ sweep with a pluggable propagation step. With the host fp32 propagator it is bitwise identical to quant.gptq (unit test). TTPropagator keeps W on the card and supports plain fp32 or a 3xTF32 split.
Host dry run on layer 0 GDN groups, CPU fp32 vs the CUDA-built G2 (this is the floor on agreement: same algorithm, different BLAS/LAPACK): proxy ratio 1.0000 on all 4 units, bfp-block agreement 99.79% (qkv, z), 100% (ab, out_proj).
Queued: the stage-1 precision probe (L0 gdn_in, L0 down_proj; propagation plus X^T X) under /tmp/ttlock.
nix run .#tt-quant -- test|ship|precision|gptq|hessians|teacher|status.

Progress (branch `tt-quant`, cdf66ac): tooling for all 5 stages is written and tested on host; card work is queued behind the MTP lock on cfx-llm2. - `tools/tt-quant/ttq.py`: GPTQ sweep with a pluggable propagation step. With the host fp32 propagator it is bitwise identical to `quant.gptq` (unit test). `TTPropagator` keeps W on the card and supports plain fp32 or a 3xTF32 split. - Host dry run on layer 0 GDN groups, CPU fp32 vs the CUDA-built G2 (this is the floor on agreement: same algorithm, different BLAS/LAPACK): proxy ratio 1.0000 on all 4 units, bfp-block agreement 99.79% (qkv, z), 100% (ab, out_proj). - Queued: the stage-1 precision probe (L0 gdn_in, L0 down_proj; propagation plus X^T X) under /tmp/ttlock. - `nix run .#tt-quant -- test|ship|precision|gptq|hessians|teacher|status`.

Grok commented

2026-09-25 10:31:54 +02:00

Author

Owner

Stage 1 measured on hardware (one P150, real L0 H and W8; record bench/runs/tt-quant-precision-20260925T080826Z.jsonl on tt-quant, commit 5105001). Relative error of the GPTQ propagation E1@U vs fp64: host fp32 2e-7; device fp32/HiFi4 1.1-1.2e-3; bf16/HiFi4 2.3e-3; 3xTF32 5.5-5.9e-4; fp32/HiFi2 6.6-7.5e-3. Device fp32 add/subtract is bit-exact vs host. So fp32 operands do not reach the FPU at TF32, and the right-looking run's 24-70% block agreement comes from matmul error, not the eltwise op. X^T X with bf16 activations, HiFi4 and fp32 acc: 5.9e-5. Next (queued behind the MTP lock): exact-bf16 3-way split (6xbf16, 6 products) on the probe, then GPTQ of L0 and L3 with tt-left-6xbf16 vs the host-propagation fallback (cpu-float32, only X^T X on the card), then the TT Hessians. origin/master (PR #68) is merged into tt-quant.

Stage 1 measured on hardware (one P150, real L0 H and W8; record bench/runs/tt-quant-precision-20260925T080826Z.jsonl on tt-quant, commit 5105001). Relative error of the GPTQ propagation E1@U vs fp64: host fp32 2e-7; device fp32/HiFi4 1.1-1.2e-3; bf16/HiFi4 2.3e-3; 3xTF32 5.5-5.9e-4; fp32/HiFi2 6.6-7.5e-3. Device fp32 add/subtract is bit-exact vs host. So fp32 operands do not reach the FPU at TF32, and the right-looking run's 24-70% block agreement comes from matmul error, not the eltwise op. X^T X with bf16 activations, HiFi4 and fp32 acc: 5.9e-5. Next (queued behind the MTP lock): exact-bf16 3-way split (6xbf16, 6 products) on the probe, then GPTQ of L0 and L3 with tt-left-6xbf16 vs the host-propagation fallback (cpu-float32, only X^T X on the card), then the TT Hessians. origin/master (PR #68) is merged into tt-quant.

Grok commented

2026-09-25 15:12:12 +02:00

Author

Owner

Status after the cfx-llm2 reboot: the 6xbf16 chain had already finished every stage before the reboot (all exit 0), so nothing needed a relaunch. Stage 4 proof is queued on the mesh slot. Results:

6xbf16 (exact 3-way bf16 split, 6 products): P_rel_fro 2.7-3.4e-4, flip rate 1-3e-5. That is about 4x better than fp32 operands but 30x short of the 1e-5 target.
Synthetic FPU probe (bench/runs/tt-quant-fpu-*): the HiFi4 multiplier is exact (one product per output, any m<=8 bits). Summing products is not. Random bf16 x bf16 dot products carry 3.1e-4 relative error, flat in K from 32 to 2048, and ones x bf16 carries 6e-5. The floor sits in the FPU's inner-product adder, so no operand split reaches 1e-5 on this path.
GPTQ, L0+L3, 14 units, CUDA Hessians: tt-left-6xbf16 proxy is 0.9997-1.0017x CUDA G2 and cpu-float32 (host propagation) is 1.0000-1.0003x. Both pass the 1% gate on all units. Block agreement vs CUDA is 32-82% for 6xbf16 and 62-100% for host (the CPU floor).
Wall time on cfx-llm2 (4 cores): per layer 6xbf16 313 s (L0) / 602 s (L3, cold kernel compile) vs host 236 s / 223 s. Warm device propagation beats host propagation (2 s vs 18 s on down_proj), but host Cholesky and rounding dominate.
Pick: host propagation (cpu-float32). It is faster end to end, passes the gate, and matches CUDA's arithmetic. The cards do the Hessian XT X.
Stage 3: H collected on the TT 1x2 mesh (prod TP=2 eager prefill, 256x2048 tokens, 8 groups over L0 and L3). The run took 323 s of forward (25 s in taps) plus 130 s of model build; on CUDA the same step is about 20 s/layer on 4x3080. Comparison vs the CUDA H is running now.
Runner: moved to ttx per-card slots (card0 for 1-card work, mesh for PROD=1); the legacy lock is not taken any more. Commits up to 0c2e5bf on tt-quant.

Status after the cfx-llm2 reboot: the 6xbf16 chain had already finished every stage before the reboot (all exit 0), so nothing needed a relaunch. Stage 4 proof is queued on the mesh slot. Results: - 6xbf16 (exact 3-way bf16 split, 6 products): P_rel_fro 2.7-3.4e-4, flip rate 1-3e-5. That is about 4x better than fp32 operands but 30x short of the 1e-5 target. - Synthetic FPU probe (bench/runs/tt-quant-fpu-*): the HiFi4 multiplier is exact (one product per output, any m<=8 bits). Summing products is not. Random bf16 x bf16 dot products carry 3.1e-4 relative error, flat in K from 32 to 2048, and ones x bf16 carries 6e-5. The floor sits in the FPU's inner-product adder, so no operand split reaches 1e-5 on this path. - GPTQ, L0+L3, 14 units, CUDA Hessians: tt-left-6xbf16 proxy is 0.9997-1.0017x CUDA G2 and cpu-float32 (host propagation) is 1.0000-1.0003x. Both pass the 1% gate on all units. Block agreement vs CUDA is 32-82% for 6xbf16 and 62-100% for host (the CPU floor). - Wall time on cfx-llm2 (4 cores): per layer 6xbf16 313 s (L0) / 602 s (L3, cold kernel compile) vs host 236 s / 223 s. Warm device propagation beats host propagation (2 s vs 18 s on down_proj), but host Cholesky and rounding dominate. - Pick: host propagation (cpu-float32). It is faster end to end, passes the gate, and matches CUDA's arithmetic. The cards do the Hessian XT X. - Stage 3: H collected on the TT 1x2 mesh (prod TP=2 eager prefill, 256x2048 tokens, 8 groups over L0 and L3). The run took 323 s of forward (25 s in taps) plus 130 s of model build; on CUDA the same step is about 20 s/layer on 4x3080. Comparison vs the CUDA H is running now. - Runner: moved to ttx per-card slots (card0 for 1-card work, mesh for PROD=1); the legacy lock is not taken any more. Commits up to 0c2e5bf on tt-quant.

No labels

human-approved

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack#67

No description provided.

Rows
Columns