The 35B MoE's NPU packing: 5,120-byte tiles, 4,736-byte rows, and an A/B interleave

The crash that started it

We want the 35B-class models on the NPU — starting with Qwen3.6-35B-A3B (the qwen3_6_moe_npu class, 673 tensors, 40 layers, vocab 248320). The blocker was not the NPU: AMD’s own runtime crashes loading this model. load_weights SIGSEGVs inside qwen3_6_reorder_cpy with a memcpy length of 0xF5E56C80 (≈−169 MB — an unsigned wrap). The gdb evidence is in the docs. (A second, unrelated wall: the qwen3_npu class rejects intermediate size 512 — the 35B is a MoE class with different checks, and our first harness instantiated the wrong one.)

So the plan changed: don’t wait for the loader — decode how the runtime packs the weights onto the device from the artifacts it leaves behind (instruction streams, TXN buffer descriptors, memory descriptors), and pack them ourselves.

The map

The layer kernel ABI is the same 5-BO shape as the 0.6B: (opcode, instr, ninstr, bo0..bo4) = act, weights, i5, i6, kv/state. The NPU2 buffer descriptors carry device addresses that are deterministic per allocation order — which is exactly why the 0.6B’s ELFs run byte-exact: our BOs land where the runtime’s would. The 35B layer-0 map:

region	device address	contents
activations	0x40000000 (1 GB)	6 refs at 16 MB
kv / state	0x2000000 (32 MB)	632 refs
MoE weights (base)	0xe000000 (224 MB)	550 refs
per-expert blocks	0x1bc00000–0x1ca00000 (444–459 MB)	4 MB steps
linear-attn state	0xc0000000 (3 GB)	560 refs

And from the descriptor walk (each descriptor is 8 words; the offset lives at word 8) — layer 0’s weight BO:

tensor	offset	size
up_exps_proj	0x0	155,189,248 (32768 × 4736)
gate_exps_proj	0x9400000 (148 MiB)	155,189,248
down_exps_proj	0x12800000 (296 MiB)	155,189,248
share_up / share_gate / share_down	0x1bc00000 / 0x1bc94000 / 0x1bd28000 (444–445 MiB)	—
self_attn.gate_proj	0x1c6fc000	—
linear_attn.qkv_proj	0x1bdbc000	2304 × 5120 tiles
moe_router / shared_expert_gate	0x3000 / 0x2000	12 KiB / 8 KiB

Three expert tensors dominate: 465 MB per layer before anything else fits.

The puzzle: 5,120-byte tiles, 4,736-byte rows

Here is the mismatch that cost the most. The GGUF file stores each expert tile as 5,120 B — [512 B scales][512 B zeros][4096 B packed]. The runtime’s in-memory rows are 4,736 B. Comparing the two: the runtime trims each tile to tile[0:4736], dropping the last 384 B of the packed payload — then interleaves rows in 16-row blocks (75,776 B):

out[o] = trimmed[o/2 + 8·(o%2)]

i.e. [A0, B0, A1, B1, …, A7, B7] where A is the first 8 rows and B the next 8 — the A/B half-interleave visible in the runtime’s disassembly as two memcpys of 4,736 B with sources 37,888 B apart. Per expert tensor that is 32,768 rows × 4,736 B = 155,189,248 B = exactly 2,048 blocks — which is also exactly the descriptor’s gate_exps offset delta (up at 0 → gate at 148 MiB). The block structure even matches the layer TXN weight BDs: 18,944-B reads (4 rows) at 75,776-B strides (16-row blocks).

The proof it is right: tools/verify_moe_reorder.cpp calls the runtime’s own qwen3_6_reorder_cpy on real up_exps tiles and compares byte-for-byte — PASS, row 0 == tile 0 [0:4736], row 1 == tile 8 [0:4736], row 2 == tile 1 [0:4736].

Verified, then the next wall

The packer built on the formula is 100% verified: layer-6 rows 0..98303 reconstruct byte-for-byte (verify_moe_packer.py). More structure fell out along the way:

• gate_proj’s k-order is (a, a−7) A/B pairs — [232, 225, 233, 226, …] — not sequential.
• Every tensor is dtype=8 (elsize 4736) — the “8704→9216 padded” theory for the qkv family is wrong; those files are 8,704-B rows with a different block format ([256 B scales][8192 B int8], block-32, zero-point 0).
• Row 0 of an expert tensor crosses the tensor boundary: share_up’s 824-B tail + up’s first 3,912 B.
• The MoE layer ELFs generate without load_weights, via the exported sequence builder: linear layers 24,636 words, full-attention layers 20,566 words (every 4th layer), lm_head 260 words.

What is left, honestly: the engine’s 35B weight-packing + forward loop (no 35B generate() yet — the E2E milestones are the 0.6B), and the qkv/ssm_out tensors’ true layout — layer-0’s captured BO is all zeros after 486.03 MB, and a capture that survives the small-tensor crashes is still needed. The upstream reorder_cpy crash also means the runtime cannot serve as the 35B reference until ROCm ships a fix — so our byte-exact verification against its own function is currently the strongest check that exists.

The numbers

measure	value
model	Qwen3.6-35B-A3B-NPU2 (qwen3_6_moe_npu), 673 tensors / 40 layers / vocab 248320
expert tensor size	32768 rows × 4736 B = 155,189,248 B (2048 blocks) × 3 = 465 MB/layer
file tile vs runtime row	5120 B → 4736 B (trim [0:4736], drop 384 B packed tail)
reorder formula	out[o] = trimmed[o/2 + 8·(o%2)], 16-row blocks (75776 B)
verification	byte-exact vs runtime’s own qwen3_6_reorder_cpy; packer 100% (layer-6 rows 0..98303)

Related

Byte-identical: 1,000 tokens of proof against AMD’s real runtime · I reverse-engineered AMD’s NPU stack in 4 days · the full map + gdb evidence are in npu-infer/docs/35b-forward-integration.md and npu-infer/docs/35b-moe-load-crash.md on the feat branch.