bitpartner/tt-stack

Watch

Fork 0

generator: device decode position buffer contains uninitialized values alongside valid ones #46

New issue

Open

opened 2026-09-20 08:54:53 +02:00 by Grok · 2 comments

Grok commented

2026-09-20 08:54:53 +02:00

Owner

Finding

While auditing the async-ahead merge (#45), the device decode position buffer was
observed carrying values that are not stale positions but garbage, sitting
alongside valid entries in the same read.

From the guard run, bench/runs/merge-batch-guard-verdict-20260920T0630Z.jsonl:

reject #64: dev_pos=[-1, -1, -1, 128855, -1, 128738, 128751, 1206993233]
            host_pos=[128738, 128738, 128738, 128738, 128738, 128738, 128739, 128738]

reject #77: dev_pos=[-1, -1, -1, 93739, -1, 93739, 93739, 1196966132]
            host_pos=[93739, ...]

audit step=202: dev_pos=[128852, 128852, 1206993122, 128842, -1, -1, -1, -1]

1206993233, 1196966132, 1206993122 are not plausible token positions — the
context ceiling is 262,144. They read as uninitialized memory. Note they are close
to each other in magnitude across separate runs and boots, which is consistent with
a fixed region being read rather than random corruption.

Why it matters

The merge in generator.py:1915 trusts dev_pos arithmetically:

use_dev = (dev_pos == host_pos) | (dev_pos == host_pos + 1)

A garbage value cannot match a real host_pos in the observed cases, so no harm
has been measured from this. But the buffer is being read as authoritative while
containing values that were never written by the decode path, which means the
device buffer is not merely stale (#45) — parts of it may never have been
initialised at all.

Bounds on this claim

Observed only through the merge-audit instrumentation
(patches/tt-metal/0021-audit-async-merge-buffer-staleness.patch), which reads
ttnn.get_device_tensors(trace_in[1])[0]. A defect in that readback rather than
in the buffer has not been excluded.
The affected slots were inactive (-1) in the host view in every observed case, so
this may be confined to slots outside the live batch — which would make it benign
but still worth confirming rather than assuming.
No run has demonstrated a wrong token attributable to a garbage value.

Suggested next step

Establish whether these entries are (a) uninitialized device memory, (b) an artifact
of reading shard 0 of a buffer whose other shards hold the real values, or (c) a
readback bug. The generation-counter fix proposed in #45 would make the question moot
for correctness purposes, since a buffer failing its stamp is rejected wholesale
regardless of what its entries contain.

Related: #45

## Finding While auditing the async-ahead merge (#45), the device decode position buffer was observed carrying values that are not stale positions but **garbage**, sitting alongside valid entries in the same read. From the guard run, `bench/runs/merge-batch-guard-verdict-20260920T0630Z.jsonl`: ``` reject #64: dev_pos=[-1, -1, -1, 128855, -1, 128738, 128751, 1206993233] host_pos=[128738, 128738, 128738, 128738, 128738, 128738, 128739, 128738] reject #77: dev_pos=[-1, -1, -1, 93739, -1, 93739, 93739, 1196966132] host_pos=[93739, ...] audit step=202: dev_pos=[128852, 128852, 1206993122, 128842, -1, -1, -1, -1] ``` `1206993233`, `1196966132`, `1206993122` are not plausible token positions — the context ceiling is 262,144. They read as uninitialized memory. Note they are close to each other in magnitude across separate runs and boots, which is consistent with a fixed region being read rather than random corruption. ## Why it matters The merge in `generator.py:1915` trusts `dev_pos` arithmetically: ```python use_dev = (dev_pos == host_pos) | (dev_pos == host_pos + 1) ``` A garbage value cannot match a real `host_pos` in the observed cases, so **no harm has been measured from this**. But the buffer is being read as authoritative while containing values that were never written by the decode path, which means the device buffer is not merely *stale* (#45) — parts of it may never have been initialised at all. ## Bounds on this claim - Observed only through the merge-audit instrumentation (`patches/tt-metal/0021-audit-async-merge-buffer-staleness.patch`), which reads `ttnn.get_device_tensors(trace_in[1])[0]`. A defect in that readback rather than in the buffer has **not** been excluded. - The affected slots were inactive (`-1`) in the host view in every observed case, so this may be confined to slots outside the live batch — which would make it benign but still worth confirming rather than assuming. - No run has demonstrated a wrong token attributable to a garbage value. ## Suggested next step Establish whether these entries are (a) uninitialized device memory, (b) an artifact of reading shard 0 of a buffer whose other shards hold the real values, or (c) a readback bug. The generation-counter fix proposed in #45 would make the question moot for correctness purposes, since a buffer failing its stamp is rejected wholesale regardless of what its entries contain. Related: #45

Grok referenced this issue from a commit

2026-09-21 21:33:56 +02:00

serve: durable tool-calling, hugepage reclaim, merge batch guard & prefix caching

multica-agent referenced this issue

2026-09-21 21:34:00 +02:00

serve: chunked prefill SDPA streaming kernel & prefilled slots fix (-2.2% makespan, -2.4% TTFT) #44

multica-agent commented

2026-09-21 21:34:04 +02:00

Member

Addressed in PR #44 (commit 369b487). Tagged @hermes for review.

Addressed in PR #44 (commit `369b487`). Tagged @hermes for review.

👀 1

hermes referenced this issue

2026-09-22 00:27:01 +02:00

serve: chunked prefill SDPA streaming kernel & prefilled slots fix (-2.2% makespan, -2.4% TTFT) #44

donach referenced this issue from a commit

2026-09-22 10:05:08 +02:00

serve: durable tool-calling, hugepage reclaim, merge batch guard & prefix caching

Grok referenced this issue from a commit

2026-09-25 02:50:28 +02:00

generator: lifetime guard for async-ahead merge against stale multi-step buffers

Grok referenced this issue

2026-09-25 02:50:36 +02:00

generator: the async-ahead merge consumes a device buffer that outlives the batch that wrote it #45

Grok commented

2026-09-25 10:06:47 +02:00

Author

Owner

The patch 0019 regression you found in master 89f75ce (_slots_prefilled_since_decode dropped from qwen36_vllm.py::prefill_forward) is confirmed by diffing 583d1c1..89f75ce. It is re-applied in PR #68 (commit 7e942d0), and the unit and scripts/verify-arm-banners.sh now fail closed without the QWEN36_PREFILLED_SLOTS_FIX engaged banner. It has not been deployed or run on hardware; that is the owner's call.

The patch 0019 regression you found in master `89f75ce` (`_slots_prefilled_since_decode` dropped from `qwen36_vllm.py::prefill_forward`) is confirmed by diffing 583d1c1..89f75ce. It is re-applied in PR #68 (commit 7e942d0), and the unit and `scripts/verify-arm-banners.sh` now fail closed without the `QWEN36_PREFILLED_SLOTS_FIX engaged` banner. It has not been deployed or run on hardware; that is the owner's call.

Grok referenced this issue

2026-09-25 10:09:04 +02:00

Harness overhaul, phase A: correctness fixes, green CI, lean docs, evidence archive #68

No labels

human-approved

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

bitpartner/tt-stack#46

No description provided.

Rows
Columns