MTP on ThinkingCap-Qwen3.8-27B-FP8 (TP=2): inventory, per-row B=4/8 design, go/no-go, phased gates #63
Labels
No labels
human-approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
bitpartner/tt-stack#63
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Tracking issue for ledger entry 0aaaaaaaaaaaaaaaaaa (owner, 2026-09-24): serve bottlecapai/ThinkingCap-Qwen3.8-27B-FP8 (rev
e55ba7f6) and get MTP self-speculative decoding working on it, on 2x P150a (TP=2). This reverses 0aaaaa ("MTP IS DROPPED"). Branch:mtp-thinkingcap. Related: #61, #62 (see the end).Labels used below: MEASURED (cites
bench/runs/*), THIRD-PARTY (someone else's number, not reproduced here), ESTIMATE (arithmetic done for this issue; not a result). House rule: extrapolations here have been wrong by +49% and +105%. Nothing labelled ESTIMATE may be used for ranking until the P1 probes land.1. Inventory: what exists, where it lives, what was measured
The code exists, but only upstream, and only at B=1. tt-metal PR #55548 (atupe-tt) is in our flake pin (
flake.nix->/home/orchestra/ttm-build/src). It is also baked into thevllm-tt:mtpimage on cfx-llm2 (ttnn 0.75.0rc10 at96f3f041,nix/mtp-pr55548/).mtp.*: fc + 1 full-attention layer + 3 norms; shares embed + lm_head; own paged KV)qwen36/tt/mtp.py(248 lines)tt/spec_decode.py(831 lines)model.verify_traced,attention/tp.py_SPEC_SDPA_L1_FIT(T in {4,8,12})bench/runs/dspark-verify-hw-20260911T131537Z.jsonl: the head-packed K+1 SDPA reproduces sequential decode on our cards (PCC 0.99999, K in {1,3,7}). MEASURED for correctness only; no timing.gdn/tp.py_verify_fullbatch+commit_verify_slot(idx)fused_recurrent_gated_delta_rule(..., output_per_token_state=True)->[B,T,Nv,Dk,Dv]. The live state is never written speculatively. The commit then copies slotidxintorec_statein place, and is traced once per idx. There is no rollback. The old "research-grade blocker" inDECODE-OPTIMIZATION.mdis superseded.spec_decode._commit(mi)mifor all 48 GDN layers.generate()takes one prompt, andpis a scalar. The batch axis is spent on the K+1 candidates. This is B=1 by construction (bench/runs/mtp-batch-structural-20260912T200000Z.jsonl).patches/prod-local-metal/,vllm-tt:k2patches/tt-metal/0006dropsmtp.*at load. The spec tree has zero production perf flags (CONV_GATES / PACKED_QKV / FUSED_INPLACE / LOGITS_RM / device argmax). The two trees are disjoint (50c6918).platform.py:1473assertsnot speculative_config.model_runner.pyassumes a fixednum_out_tokens. There is no proposer and no rejection sampler.lane_scheduler.pyalready forwardsupdate_draft_token_ids/num_invalid_spec_tokens.568afb3a, invllm-tt:k2)qwen3_5_mtp(model_executor/models/qwen3_5_mtp.py) andmethod="mtp"inconfig/speculative.py. Per-row hybrid spec is already modelled:v1/attention/backends/gdn_attn.pycarriesspec_state_indices_tensor [batch, num_spec]andnum_accepted_tokens [batch]. That is exactly the per-row design needed below. The rejection sampler is Triton, which is unusable here (VLLM_TARGET_DEVICE=empty), but greedy acceptance is a host prefix match.What was measured (all B=1, none on ThinkingCap):
bench/runs/mtp-ab-failure-20260912T193000Z.jsonl) loaded the head (mtp=True, model load 1215 s). It then died in a JIT kernel build (writer_unary_stick_layout_wh_multicorestatic assert) on a PLAIN arm. The cause, broken image vs a stale sharedTT_METAL_CACHEentry, was never settled./var/lib/models/tt-cache/P300/tensor_cache_bfp8_mesh1x2/mtp/(2026-09-11) holds the Qwen 27B MTP head converted for TP=2.metal-mtp/is that image's kernel cache.tt-cache/mtp/(2026-09-05) is a different, MoE-shaped head (it hasse_router), not ours.bench/runs/mtp-draft-token-20260911T144311Z.jsonl). This is a pessimistic bound, since proxy logits are flat.bench/runs/verifier-cost-model-20260919T0500Z.jsonl): V(T) ~= step + 0.68*(T-1) + attn(T) + 3 ms commit. Its inputs are THIRD-PARTY or standalone, not in-model on our tree.dspark-acceptance-rate-20260911T153249Z.jsonl). That was a different drafter, not MTP.lookup-draft-acceptance-20260919T0400Z.jsonl). Workload regime is the dominant term.New today (MEASURED, no device):
bench/runs/thinkingcap-ckpt-contract-index-20260924T0935Z.jsonl: ThinkingCap vs our Qwen snapshot. Base key set is identical (1584 keys). The arch config is identical and the FP8 scheme is identical. The 15mtp.*weights carry noweight_scale_inv. Header-level dtype/shape checking runs once the download completes.LazyFP8StateDict) dequantises onlyfloat8_e4m3fntensors and passes bf16 through, so the bf16 head needs no loader change in the spec tree. Verified on the Qwen snapshot bybench/mtp/check_mtp_load.py(host only). The ThinkingCap run is queued behind the download.2. Design for B=4/8: per-row accepted lengths
The goal is to replace the scalar
miwith a per-rowm_b. Everything is keyed to one constraint: B·(K+1) <= 32 verify rows. Decode matmuls already pad M to one 32-row tile, so up to 32 rows the weight read and the matmul intercept are flat. Beyond 32, a second tile costs ~8-10 ms (n1-matmul-read-compute-split-20260912T174500Z.jsonl). B=8 => K<=3; B=4 => K<=7.p_b+1..p_b+T. It must be the packed spec SDPA extended with a batch axis, so each sequence's KV is read once, not T times. KV writes go row by row (alias_kv_writeexists). Rejected slots need no rollback, because the per-row frontier advances only bym_b+1.[B,T,Nv,Dk,Dv]. The conv window becomes[B, K-1+T, C]instead of[1, ...].[B]is refreshed from host outside the trace. One gather per layer selectsstates[b, m_b]and the conv window rows[m_b, m_b+K). ESTIMATE: ~1-2 ms over 48 layers at B=8, versus 3.2 ms scalar traced commit at B=1 (THIRD-PARTY, upstream note).m_b. This needs a per-row initial-state index in the kernel. It is the best long-term option and upstreamable.beta=0, g=0beyondm_b:S*exp(0)+0leaves the state bit-unchanged. It costs a second recurrence pass (~0.85+0.68·T ms at B=1, THIRD-PARTY Thatch). It saves the history memory, which is ~2.3 GiB/chip fp32 at B=8,T=4.m_bas the longest matching prefix. The gate is batch-wide greedy-only: any sampled or penalised row sends the whole step to plain decode, the same rule as device argmax.speculative_config={"method":"mtp","num_speculative_tokens":K}. Drop the platform assert. The runner returns per-row variable token lists plusdraft_token_idsinModelRunnerOutput, with drafting done in-model on TT. Rejection is a host greedy prefix match.3. Go / no-go per batch size: ESTIMATE
bench/mtp/go_nogo_estimate.py->bench/runs/mtp-thinkingcap-go-nogo-estimate-20260924.jsonl. Model:cycle = step(B) + c_row·B·K + tile_penalty + commit(B) + K·d_draft + reseed,tau = (1-a^(K+1))/(1-a),S = tau·step/cycle.Inputs:
Ranges run from a=0.50 with pessimistic costs to a=0.80 with optimistic costs. The makespan column assumes decode after the last TTFT is the whole decode cost, which puts decode at 46-89% of these cells. That disagrees with the ledger's "prefill 74-79% of wall" and needs reconciling. Either way, prefix caching raises the decode share for agentic traffic.
Verdict (ESTIMATE):
Cheapest verify_cost_ratio probes:
max_model_len:c_row ~= (step(32)-step(8))/24. This is an upper bound: B=32 reads 32 sequences' KV and holds 32 GDN states, where a verify has 8. It is a probe only; B>8 is not a serving target.[SPEC_TIMING] verify=at T=4 and T=8 in the vllm-tt:mtp tree at B=1. This gives the in-model ratio, not the absolute ms.4. Phases, fail-closed probes and gates
tt-cache, a missing shard, or MemAvailable < 4 GB. A lazy-loader overlay avoids the image's ~54 GB eager FP8 load.mtp.*tensor vs the Qwen head. If it shipped the base head in bf16, acceptance against the fine-tuned base is at risk.bench/mtp/run-thinkingcap-b1.sh). DistinctTT_CACHE_PATH, fresh kernel cache, lock protocol, detached.bench/mtp/reduce_accept.py); a smoke on the fresh kernel cache before any arm.QWEN36_SPEC=0(default off). The production loader keepsmtp.*only underQWEN36_MTP=1, because 0006 drops it today. Gates: SPEC=0 is output-identical to production onbatch-equiv-ref-*at B=4/8 and within 0.5 ms; B=1 spec equivalence re-passes in-tree; aQWEN36_SPEC engaged K=.. B=..banner is present (arms must self-identify).method=mtp, with the MTP layer in the hybrid cache spec for prefix caching. Gates:bench/ab/sweep-32k.shwith a spec arm vs the cleared arm (and vs the prefix-caching arm), one source tree, env-only difference. Rank by makespan/TTLT, never decode ms.Risks carried
Related
Progress 1 (2026-09-24): P0b done. ThinkingCap ships the BASE Qwen MTP head, and P1a is on the cards
Checkpoint (MEASURED, no device):
bench/runs/thinkingcap-ckpt-contract-header-20260924T1015Z.jsonl: PASS at header level.mtp.*tensors are all BF16, match Qwen's shapes, and carry no scale. The head has 424.7M params.bench/runs/mtp-thinkingcap-host-load-20260924T1012Z.jsonl: PASS. The spec tree's lazy loader hands every bf16mtp.*tensor toQwen36MTPwith the right keys, dtypes and shapes. The FP8 base path still dequantises. Wall time was 4 s.Finding: the MTP head was NOT retrained with the fine-tune. I compared each
mtp.*tensor against the Qwen head:So ThinkingCap ships the base model's head in its pre-quantisation bf16 form. The head predicts Qwen's next-next token.
Finding: the base fine-tune is light.
bench/runs/thinkingcap-base-diff-20260924T1020Z.jsonlbyte-compares a sample of base tensors:embed_tokens,lm_head, the final norm and the FP8 block scales are identical.A light fine-tune plus a base head is consistent with the THIRD-PARTY accept_len 2.55/4, which was measured with this same head. Acceptance is still the P1a measurement, not an assumption.
P1a is running (
bench/mtp/run-thinkingcap-b1.sh, detached on cfx-llm2, host lock held asmtp-smoke:*):tt-cache/thinkingcap-e55ba7f6(stamped with the checkpoint). The fresh kernel cache ismetal-mtp-thinkingcap-20260924T094156Z.reduce_accept.py.No speed numbers yet (no PP tok/s or TG ms/token from this work).
Progress 2 (2026-09-24 12:25 CEST): P1a attempt 1 was stopped externally, so there is no measurement yet
bench/runs/mtp-thinkingcap-b1-p1a-attempt1-20260924T1023Z.jsonlWhat worked:
mtp/, is fully built attt-cache/thinkingcap-e55ba7f6.What stopped it:
tenstorrent-serving.service. The service's card probe failed because our container held the cards; that unit does not honour/tmp/ttlock.systemctl stop tenstorrent-serving.service, and our container got SIGTERM. dockerd force-killed it 60 s later (exit 137).This is not an image verdict.
run-chain.shhad mislabelled it as one, and that is fixed in8b8c9cc. The 2026-09-12 JIT question is still open, because the kernel involved is built at first decode, which this run never reached.Blocker: card contention with the production serving deploy. I did not retry, to avoid fighting it. Two consequences:
sudo tt-smi -r.cd /var/lib/models/mtp-thinkingcap && setsid nohup bash run-thinkingcap-b1.sh > driver-$(date -u +%Y%m%dT%H%M%SZ).log 2>&1 < /dev/null &The tensor cache is warm now. Expect a few minutes of load, then roughly 30-40 min for the smoke plus the 20 arms.
A suggestion for the serving unit: have
tt-probe-cards/ExecStartPrewait on/tmp/ttlockrather than failing into start-limit.Still no speed numbers from this work (no PP tok/s or TG ms/token).
P1a attempt 2 (2026-09-24 10:26-10:34Z): result is FAIL_IMAGE_BROKEN, with no numbers
Settled: the
vllm-tt:mtpimage is broken; the shared kernel cache is not. I ran the stock demo smoke (traced_128, SPEC=0, no custom code) on a fresh kernel cache. It failed in the kernel JIT with the same error as 2026-09-12:compile_time_args.h:27 static assertion failed: Index out of rangewhile buildingwriter_unary_stick_layout_wh_multicore. The ttnn 0.75 wheel's device headers don't match the kernel sources baked into/opt/tt-metal-src, so the pins did not move as a set./var/lib/models/tt-cache/thinkingcap-e55ba7f6is built.tenstorrent-serving.serviceis enabled but failed. A deploy-rs activation could restart it. I noted this and did not touch it.Incident, now fixed in the harness:
head -clog cap./tmp/ttlockwhile our container still held the cards.p150-autoresearchtook the lock.docker stop -t 150, 10:31:53-10:34:24Z). p150-autoresearch may have started before our container was gone.6e9731a:release()now stops the container before it frees the lock.run-chain.shnow takes its image verdict from the log, not the exit code.Record:
bench/runs/mtp-thinkingcap-b1-p1a-attempt2-20260924T1031Z.jsonlNext: rebuild the MTP image with the ttnn wheel and kernel sources from one tt-metal commit, built locally and shipped. The candidate is the ttm-build flake pin, which already has #55548 merged with main. The alternative is to port
spec_decode/mtponto the ttnn 0.79 production tree. After that, rerun P1a.Root cause of the broken MTP image found at file level; vllm-tt:mtp2 fix queued
I ran
nix/mtp-pr55548/verify_onecommit.pyinside the existingvllm-tt:mtpas a negative control (MEASURED,bench/runs/mtp-image-onecommit-negctl-20260924T1044Z.jsonl):/opt/tt-metal-srcis not 96f3f041. 754 files differ from the wheel and 1256 are missing.writer_unary_stick_layout_wh_multicore.cpp, the kernel that hit the static assert, is among the differing files. The old Dockerfile swapped only the wheel and the qwen36 directory.Fix:
nix/mtp-pr55548/Dockerfile.onecommitbuildsvllm-tt:mtp2on top ofvllm-tt:mtp.tt_metal/andttnn/cpp/.docker savewas not an option.Status: the driver is running detached on cfx-llm2 and is waiting for
/tmp/ttlock, currently held by p150-autoresearch. It will not preempt. Once it has the lock it will, in order:vllm-tt:mtp2and run the host load check, both inside the lock;Commit
9acc5c6pushed tomtp-thinkingcap.Smoke result on vllm-tt:mtp2: the JIT problem is fixed, but a TP=2 prefill L1 clash is next
vllm-tt:mtp2built in 19 s. It is a copy-only layer, built inside the lock. The one-commit check PASSED: the archive, the wheel and the JIT tree all match 96f3f041 with 0 mismatches. The host load check also passed.tt-smi -runder our own lock, as the protocol says.Statically allocated circular buffers in program 100 clash with L1 buffersin the GDNttnn.conv1don a[1,2051,1,5120]input. The PR hardcodesCHUNK=2048, which was validated on QB2 (TP=4). At TP=2 each device holds twice the conv channels, so the L1-resident conv overflows.QWEN36_PREFILL_CHUNKoverride. I adopted it as an overlay ondemo/text_demo.pywith default 512.bench/runs/mtp-thinkingcap-b1-p1a-attempt3-4-20260924T1055Z.jsonl. Commit1d42736.Card slot: the owner gave the next slot to the ThinkingCap baseline agent, which holds the lock now (
tc-baseline:…). Our driver is queued behindafter-owner.sh. It starts only aftertc-baseline:has taken the lock and released it, and it will not preempt.No-MTP baseline for ThinkingCap and base Qwen on the production arm is in #65 (branch
thinkingcap-baseline). Serving speed is the same for both checkpoints: TG 48-62 ms/token, prefill 2.25-3.25k tok/s. With thinking on, ThinkingCap's GSM8K completions are 35% shorter at the median and 52% shorter on the mean. One finding affects this issue: the deployed argmax tree needs the 0019 slot fix, or it fails infbench stage-3 concurrency.P1a blocked: PR #55548's GDN prefill does not run at TP=2 on the P300 pair
I ran after the tc-baseline agent had taken and released its slot. Five attempts, each a stock
text_demo traced_128run with SPEC=0 invllm-tt:mtp2, recordbench/runs/mtp-thinkingcap-b1-p1a-attempt5-9-20260924T1410Z.jsonl:ttnn.conv1dchunk_gated_delta_rulettnn.copy. Dispatch is async, so the host blocks at the next op that waits on the queue, not at the op that hungTIMEOUT: device timeout in fetch queue wait. The timeout names no op. Triage needsttexalens, which is not in the imageAfter each hang the container was stopped with
docker stop -t 150(each ran to SIGKILL), thentt-smi -runder our lock. The cards are reset and the lock is free. No host OOM or pinned-page leak.Conclusion:
Options:
dump_running_operations.pyagainst the live hung process.Commit
d37e202. The overlay keeps all workarounds as env switches, andEXTRA_ENVis available for diagnostics. Skillsmtp-ttandtt-card-slotare in the repo (4e1b1e9, trap 8 added ind37e202).Status summary 2026-09-24 (before the triage and bisect runs)
1. The old MTP image was broken; fixed.
vllm-tt:mtpshipped the PR #55548 CI wheel (96f3f041) but compiled its kernels from the 0.79 production source tree. 754 files differ from the wheel and 1256 are missing, which gave thecompile_time_args.h:27static assert.vllm-tt:mtp2(nix/mtp-pr55548/Dockerfile.onecommit) takes the compile tree from the git archive of 96f3f041, overlaid with the wheel's own sources.verify_onecommit.pyfails the build on any mismatch. The wheel matches the archive on all 4634 shared files, and the tree matches the wheel with 0 mismatches. The same check fails on the old image, as it should (negative control).2. Prefill conv hits an on-chip memory (L1) clash at TP=2; worked around.
ttnn.conv1dreports "static circular buffers clash with L1 buffers".QWEN36_GDN_CONV1D=0switches to the simpler MAC filter the code already uses elsewhere.3. The TP=2 prefill then hangs the device; open.
TT_METAL_OPERATION_TIMEOUT_SECONDSturns it into a TIMEOUT but names no operation.4. Still no numbers. No acceptance, verify_cost_ratio, token identity, prefill tok/s or decode ms/token for either arm.
5. Go/no-go per batch size is still an ESTIMATE (
bench/runs/mtp-thinkingcap-go-nogo-estimate-20260924.jsonl):6. Next, per the owner:
ttexalensto the image, keeping the one-commit check passing, and run the upstream triage tool against a live hang to name the operation.Skills:
.claude/skills/mtp-tt,.claude/skills/tt-card-slot. Branchmtp-thinkingcap@d37e202.Hang diagnosis: the fused GDN out-proj
matmul_reduce_scatter_asyncdeadlocks on the 1x2 meshOne card slot, two methods, and they agree (MEASURED,
bench/runs/mtp-thinkingcap-hang-triage-20260924T1942Z.jsonl, raw outputs in the directory of the same name):vllm-tt:mtp2-triage(mtp2 plus the tree's own triage requirements, binary wheels; the one-commit check still PASSES).dump_running_operationsreports op 1560,MatmulReduceScatterAsyncDeviceOperation, on both devices, 36 cores:[1,1,512,3072]fp32 x[3072,5120]bf8, producing[1,1,512,5120]and RS[1,1,512,2560].dump_callstacksshows every worker parked inreduce_scatter_minimal_async: the reader waits onout_ready_sem(reader.cpp:416), the ring_reduction compute waits on CB tiles, and no matmul kernel is still running.QWEN36_OP_SYNC=1): the last op started without completing is[OPSYNC] B 2836 ttnn.experimental.matmul_reduce_scatter_async tt/tp_common.py:553, the GDN out-proj of layer 0. Serialising the ops does not make the hang go away, so this is a deterministic deadlock, not a race.ttnn.copywere only where the async host happened to block next.vllm-tt:k2 gdn/tp.py:197-205already turns this fusion off for 2-device meshes with a unit dimension ("deadlocks there in every offset/links/topology config probed"). It uses_row_proj+tt_all_reduceinstead. PR #55548 was branched without that gate.QWEN36_GDN_OUT_MMRS=auto(default) applies the production gate;1reproduces the hang,0forces the fusion off. The fused chunk op is back to the default.matmul_reduce_scatter_asyncdeadlocks on a 1x2 Blackhole mesh (Linear topology, num_links=2, grid (8,8), rs_offset (0,8), M=512, K_local=3072, N=5120, fp32 out). The same inputs pass when the matmul andreduce_scatter_minimal_asyncrun unfused. The production comment adds a likely cause: on the p150a 1x2 worker grid, that placement puts 14 of the 36 reduce-scatter cores off-grid, and the op accepts it without validating.Next: the full B=1 P1a chain (smoke, then the 20-arm matrix) is queued on
vllm-tt:mtp2-triagewith the gate on. It is waiting for the lock, which the DSpark agent currently holds. Commit3de2491.First PR #55548 run completes on the P150 pair: plain B=1 smoke PASS
text_demo traced_128(SPEC=0) PASSES at TP=2. This is the first time the PR stack has run to completion on our pair.bench/runs/mtp-thinkingcap-b1-p1a-smoke-pass-20260924T2049Z.jsonl,…-attempt10-…jsonl.Now running: the 20-arm B=1 MTP matrix (
thinkingcap-b1-20260924T204859Z): plain vs spec at K=3 and K=7, over agentic_edit / code / prose / QA. It will give acceptance per regime, verify_cost_ratio, token identity, and prefill/decode speed per arm.P1a (B=1): the full matrix completes at TP=2, but token identity fails
Run
thinkingcap-b1-20260924T210159Z, imagevllm-tt:mtp2-triage(PR #55548 @96f3f041), 20/20 arms ok. All numbers are MEASURED. Record:bench/runs/mtp-thinkingcap-b1-p1a-matrix-20260924T2135Z.jsonl(plus.reduced.json/.arms.jsonl), commitfbc7900.One new TP=2 fix was needed first. The spec verify SDPA (
spec_multi_pos_tiles) requires 1 KV head per device. That holds at TP=4, but TP=2 has 2. The overlay now falls back to the legacy B=T verify call.Acceptance (committed tokens/iteration, count_control excluded)
Speed (B=1; the PR tree, not production, so compare ratios only)
Gate: greedy token identity FAILS on 7/8 arms (code passes at both K)
Hang diagnosis: unchanged from the earlier comment (fused GDN out-proj MMRS deadlock at 1x2; production gate applied in the overlay).
Token-identity gap study: queued
We need to know whether the 7/8 identity failures are bf16 near-ties or a real bug. That answer feeds the owner's tolerance decision. Harness commit
7b80e4a.QWEN36_LOGIT_PROBE=1, both paths record the top-16 logits of every row they emit a token from. Plain decode reads its trace logits; spec reads the verify[T, vocab]rows.reduce_gap.pyreports, for both paths, the margin of its own token over the other path's token.Token-identity gap study: the divergences are bf16 near-ties, not a bug (MEASURED)
thinkingcap-gap-b1-20260924T214536Z, finished 22:23Z, before the API outage.bench/runs/mtp-thinkingcap-b1-identity-gap-20260924T2223Z.jsonl(plus.reduced.jsonand.arms.jsonl.gz), commit9bbd44d.Result
Classification
The 5 flagged by the strict rule have plain margins of 0.375–0.5 (3–4 ulps). The tolerance is the owner's decision; this data sets its scale.
Localisation
The classification did not require it. One swap failed, one never ran:
commit_verify_slotcopies into_conv_win_buf, which only the full-batch path allocates. It fails as a host TypeError; the device was not affected.The main arms were not affected by the outage.
MTP on bfp4 (G2): feasibility done; the G2 vs P A/B is running
(a) Two single-card instances: PASS (MEASURED)
Record:
bench/runs/two-card-coexistence-20260925T0625Z.jsonl.--device /dev/tenstorrent/N(the owner's recipe), and separately both nodes mapped withTT_VISIBLE_DEVICES=0/1. Both variants were run under our lock.HugePages_Freestayed at 4 of 4 throughout; these containers do not use the host hugepage pool.(b) TP=1 full-model MTP: NOT VIABLE without new code
Record:
bench/runs/mtp-tp1-feasibility-20260925T0640Z.jsonl.prefill_for_spec,verify_forwardandverify_tracedallassert num_devices > 1.G2 on the PR tree
QWEN36_ATTN_GDN_BF4gate into the overlay's PR-tree loaders (5 attention and 4 GDN weight sites, the same as the bfpq copies).QWEN36_ATTN_GDN_BF4_MTP=1opts in), so the drafter is identical across arms.engagedplus 1exemptbanners; P must print none.g2-ab.shholds one lock across G2 then P, with a separateTT_CACHE_PATHfor each:mtp-G2-e55ba7f6(a cold build) andthinkingcap-e55ba7f6.Commit
389b170.MTP on the bfp4 quant: G2 vs P, B=1, TP=2 (MEASURED 2026-09-25)
Setup
g2ab-20260925T064042Z. Both runs held one lock and each used its own tensor cache.Acceptance (committed tokens per iteration, 20 prompts)
Speed at B=1 (timed arms, one run per prompt)
PP is prefill tok/s. TG is ms/token. Makespan is derived as TTFT + 256 × ms/token; it is not an end-to-end wall.
Identity (owner B=1 rule, unchanged: near-tie iff swing ≤ the fused/unfused control p99)
B=4/8: not measurable, because there is no per-row commit.
Records:
bench/runs/mtp-g2-vs-p-b1-20260925T0640Z.{jsonl,compare.json,gap-G2.json,gap-P.json,arms-*.jsonl.gz}.TP=1 (single card): the code is implemented, committed and pushed (
QWEN36_TP1=1, the TP stack at divisor 1). The hardware sessiontp1-session.shis queued behind the current lock holder (tt-quant:gptq), and there is no TP=1 number yet.TP=1 MTP (63b) is reopened as engineering work, not a limit.
mtp-tp1-feasibility-20260925T0640Zis CODE-FACT + ESTIMATE: the blockers are our ownnum_devices > 1asserts. It is also tracked under the 1-card target, #69.Reopened under the new rule (owner, 2026-09-25): a closure needs an on-hardware record at the stated scope; estimates, code-reads and third-party numbers close nothing. See docs/LEDGER.md "Reopened" (PR #68).
TP=1 MTP: interim status (2026-09-25 10:10Z). No new speed number in this update.
"Decode on one card is broken": there is no decode failure on record yet. No TP=1 run has reached decode.
g2-tp1-proberuns (cfx-llm2:/var/lib/models/ttx-runs/ttx-g2-tp1-probe-smoke-20260925T083748Z and thettx-ab-...-20260925T090800Z-r{0,1}-{A,B}runs) are load-only. Their phases are imports, open, ttnn_matmul, model_load, hold and done, and every one endsok: true. They show that G2 loads on one card: 20.96 GB of 33.1 GB DRAM, VmHWM 5.6–5.8 GB, and 32–34 MB RSS after load. They never prefill or decode.nlp_concat_headsover 48 heads needed 1.68 MB.TP1_L1_FIT: blocked prefill outputs and grouped concat-heads), and neither affects TP=2. Seebench/runs/mtp-tp1-attempts-20260925.jsonl.bench/mtp/tp1-card.sh). It is waiting for MemAvailable to reach 4 GB: tt-quant's GPTQ on card 0 holds 2 GB RSS, and a TP=1 load peaks at ~5.8 GB. If decode fails there, I will report the op, file and line.TP=2 byte-identity with the TP=1 code: whole-chunk prompts are unchanged; the tail-chunk difference is pre-existing (
bench/runs/mtp-tp1-tp2identity-20260925.jsonl).Docs moved to the new layout: LEDGER
R-2026-09-25-mtp-g2andD-2026-09-25-mtp-tp1, MTP.md and STATUS.md.TP=1 decode on one card now runs: G2 plain and MTP K3/K7 on a single P150a (card 1, ttx per-card slot). Label: TP=1 iteration, not production. Record:
bench/runs/mtp-tp1-smoke-20260925T1310Z.jsonl.Why decode looked broken. No TP=1 run had ever reached decode:
g2-tp1-proberuns are load-only.The root cause of the last failure (attempt 3, 13:08Z):
ttnn.experimental.nlp_concat_heads.tp_common.py:74 concat_heads_fit, called fromgdn/tp.py:761 forward_prefill(fused out: rms_norm, then concat heads), called frommodel.py:1798 _forward_prefill_chunk_tp, during the warmup prefill trace capture.TT_THROW dataflow_buffer.cpp:2617 "static dataflow buffers ... clash with L1 buffers ... L1 buffer at 860160, CB region ends at 898048".Fix (commit on
mtp-thinkingcap).tp_common.pf_l1()returns DRAM underTP1_L1_FIT(set only at one device) and L1 otherwise. It is applied at every short-lived prefill L1 site:At TP=2 every call returns what it did before. With prefill fixed, decode ran first time, with no decode-side change.
Smoke numbers (B=1, 4096-token prompts, G2, 64 engaged / 1 exempt banners):
Next: gate (c), two TP=1 containers at once (card 0 on the s2a arms, card 1 on the s2b arms, staggered, with watchdog and mem.tsv). That run also supplies the 20-prompt data for gate (a), identity plus logit PCC against TP=2, and gate (b), K3/K7 acceptance. After that comes the solo timed s1 run for TP=1 PP and TG. The pair is queued behind tt-quant's mesh run.
TP=1 MTP on one P150a: gates measured. G2, B=1, greedy, 256 tokens, ttx per-card slots. Label: TP=1 iteration, not production; rankings stay on TP=2 E2E. Records:
bench/runs/mtp-tp1-gates-20260925T1410Z.jsonl(plus.compare.json) onmtp-thinkingcap(d2e9fea). LEDGERR-2026-09-25-mtp-tp1.Speed (card 1 solo, timed arms with the probe off), ms/token:
Gate (b), acceptance: PASSED. Committed tokens per iteration, paired over 20 prompts:
Gate (c), two TP=1 containers at once: PASSED.
bench/mtp/tp1-pair.sh).Gate (a), identity vs TP=2: NOT PASSED under the owner rule (control p99: TP=2 1.94, TP=1 1.81).
TP=2 byte-identity with this code: whole-chunk prompts are bit-identical. The 2048+47 tail-chunk prompt varies with any process change, including the old overlay with a new harness. That is pre-existing (
bench/runs/mtp-tp1-tp2identity-20260925.jsonl).Code changes this round (all gated to one device):
tp_common.pf_l1(): DRAM for short-lived prefill activations that TP=2 keeps in L1. This is the fix that got prefill through and let decode run.run-mtp-ab.sh:SLOT_OWNERmode.tp1-card.shandtp1-pair.sh: the ttx-slot drivers.Docs updated: LEDGER, MTP.md, STATUS.md, and the mtp-tt and tt-card-slot skills.