Byte-identical: 1,000 tokens of proof against AMD's real runtime
Last week ended with corr 1.000000 on one prefill shape. This is what it took to hold byte-identity through a whole generation — a validation loop that closed layer-by-layer, a rope table reverse-engineered out of the runtime’s .rodata, and the one place byte-identity does not hold, found honestly.
Milestone: our NPU decode is now bit-for-bit identical to AMD's own runtime — verified across a full 1,000-token generation, zero differences. Full technical write-up → journey.md — 09-02 session
The bar we set
Last week’s Zaya post ended with a number that sounded like a rounding: the engine’s hand-rolled NPU launcher hit corr 1.000000 against the real FastFlowLM runtime on prefill, argmax 397 == 397. This post is what it took to hold that through a whole generation — and the one place where byte-identity does not hold, which we only found by chasing it.
Why byte-identity and not “corr 0.9999”? Because once the Q4NX dequant was proven lossless (W = (q − zp) · scale, maxdiff 0.0 over the whole projection) and the weights were bit-identical to the reference, any corr < 1 is an execution difference — summation order, a missed table entry, a silently no-op’d kernel. Same instruction stream, same bytes, is the only number with nothing hiding behind it.
The validation loop closes
The prefill result validated one shape. To close the loop we diffed the whole forward chain — every layer’s activations, not just final logits — against runtime captures at every context depth. Three facts made it hard, then made it possible:
- The 28-layer “explosion” is real. After one forward the activations have std 194.4619 (range [−1216, 780]) — the earlier “std 3.39” that looked like divergence was an FP16-vs-BF16 numpy artifact in our own reference tooling.
- The runtime arms its layer kernel twice per layer. One run reproduces the capture byte-exactly; the second arm is invisible unless you look.
- One layer kernel is the whole layer — attention + MLP fused, reading 1,920 weight tiles (9.8 MB) per forward.
| check | activations | logits argmax | vs runtime |
|---|---|---|---|
| fwd1 (ctx 1, 1/2/3/28 layers) | std 194.4619 | 397 == 397 | maxdiff 0 |
| fwd2 (ctx 2) | std 19.8162 | 88 == 88 | maxdiff 0 |
| fwd3 (ctx 3) | std 25.6497 | 284 == 284 | maxdiff 0 |
generate() end-to-end on the runtime path
With the layer path validated, we wired it into the engine (RuntimeLayerEngine, NPU_RUNTIME_LAYERS=1) and ran a real generation, BOS→16 tokens. ctx1..16: maxdiff 0. Then ctx17 lit up: maxdiff 0.3399 — exactly one bf16 ULP, argmax 9695. It turned out to be a rope-table tie-boundary artifact where the engine was the more accurate side: the runtime’s table value sat 0.7% of an ULP above a rounding tie, so it rounded up one bf16 ULP, while the engine’s exact double-precision math kept the lower ULP. The remaining contexts validated on-device (ctx18..21, ~19 ms/tok). Two clean-reboot drills later, the whole chain was reproducible — the hardware state survives, the ELF set survives, the bytes survive.
The rope table that was too accurate
Chasing ULP-level diffs is where the interesting reverse engineering happens. The engine computed rope’s phi = pos · 1e6(−2j/128) in double precision — and that was the bug: the runtime keeps a hardcoded float32 inv_freq[64] table in libqwen3_npu.so’s .rodata @0x152740, and the literals are not the f32 rounding of the double formula (off by up to ~1.5e-5 relative, in a non-monotonic per-j pattern no powf/expf/sincosf chain reproduces). Decoded from disassembly of the runtime’s _rope_rms: phi = inv_freq[j] · (float)pos (float32 multiply), glibc sincosf, f32→bf16 RNE. We now embed the exact 64-float dump and use the same arithmetic.
| test | result |
|---|---|
| 40-token decode (ctx2..41) | 40/40 byte-identical, 0 ULP, 0 argmax flips |
| 63-token decode (ctx2..64, full shipped ELF range) | 63/63 byte-identical |
| multi-token prefill (ctx1..4) then decode | byte-identical |
| 200-token decode (ctx2..201) | 200/200 byte-identical |
| 1000-token decode (ctx2..1001) | 1000/1000 byte-identical, 0 ULP, 0 argmax flips |
Rope holds to phi ~800 rad through glibc’s argument reduction, and 1,000 tokens of KV growth stay exact. Two things made the ladder cheap to run: per-context ELFs now generate on demand (~0.6 ms each; full MAX_L 4096 is ~2.5 s) instead of shipping 290 MB of binaries, and capture hygiene — a CAP_NO_SYNC gate cut the interposer’s sync/wait dumps from 45 GB to 500 MB of lean i6-only captures. Same round: the decoder finally got real sampling (temperature / top-k / top-p, seeded) replacing a dead greedy stub — and one genuine bug: the RNG was reseeding every draw, so every token sampled the same stream value.
The honest footnote: batched prefill ≠ N× forward
Byte-identity held for the sequence path. Then we asked the obvious next question: is the runtime’s batched prefill (what a served session actually uses) equal to N× single-token forward? No: prefill vs forward@ctx4 logits corr 0.945, maxdiff 3.69, argmax 7829 vs 97462. The root-cause hunt went through three claims in under an hour — worth keeping because the first two were wrong:
- Claim 1: rope-table divergence. Wrong — the batched mm never advances the host i6 table, so every dump sat at pos 0 where rope is identity.
- Claim 2: two confounders (mv-vs-mm projection ULP at pos 0, rope table at pos>0). The fp64 adjudication then showed neither path is byte-correct: mm 92/1024 byte-match (834/1024 ≤1 ULP), seq 17/1024 (866/1024) — both valid bf16 pipelines, ~0.5% mean relative error.
- Claim 3 (final): the rope confounder was a magnitude artifact. The big-diff dimensions (50/115) are the largest-|K| dims (mean |K| 85.9/13.4 vs 7.5 next-largest), never rope-paired, present at pos 0; |diff| vs |K| corr 0.63–0.84. The .rodata-vs-exact error is ≤5e-8 rad in phi — physically invisible in bf16.
Final root cause: one confounder — batched-mm vs per-token-mv GEMM accumulation numerics. The engine stays byte-identical to the runtime’s sequence path at every context; a served AutoModel-chat session (batched prefill) can differ at first-token argmax on near-ties. That is runtime-internal, not an engine defect — and it is exactly the kind of caveat we keep in the docs rather than hiding.
The numbers
| measure | value |
|---|---|
| prefill logits vs runtime (token-1000 input) | corr 1.000000, argmax 397 == 397 (round 28) |
| forward chain fwd1/2/3 vs runtime | byte-identical, maxdiff 0 |
| decode through 1,000 tokens (ctx2..1001) | 1000/1000 byte-identical, 0 ULP, 0 argmax flips |
| decode speed (Qwen3-0.6B / XDNA 2) | ~15 ms/tok; on-device gen ~19 ms/tok |
| per-context ELF generation | ~0.6 ms/ELF on demand (shipped set stays 1..64, 9.9 MB) |
| capture size (CAP_NO_SYNC) | 45 GB → 500 MB |
Related
Zaya 8B Q4NX on the AMD NPU · HRX: in-process fused decode · the full round-by-round record (including the retracted claims) lives in npu-infer/docs/txn-decode-findings.md on the feat branch.