1bit engine

Where the engine is going: HRX and the NPU

2026-09-29T00:00:00Z

The engine is heading to AMD's HRX backend, whose kernels we write in Loom, plus the NPU.

A review of the Laya work¹ put it plainly: we want it on the NPU, a Vulkan version already exists, and the stack is HRX + Loom + NPU. That was right, and not only for Laya. Vulkan stays the default until HRX meets the gates in RFC #213: decode within 10% of Vulkan on the showcase models, every architecture that runs on Vulkan checked on HRX, and speculative decoding, several sequences per batch and vision on HRX. Here is what landed since the last posts, and how far HRX already is.

Laya's scorer runs on HRX#

Laya classifies each conversation as code, prose, short or a long document, and a measured policy picks the device (previous post). The classifier is a ModernBERT encoder. It now runs through ggmlc (our fork) on HRX, with Loom kernels for what the encoder needs: norms, rotate-half RoPE, GEGLU and the attention scores (#206, #208).

Scorer	Accuracy	Per decision
C++ on the CPU	95.5%	538 ms
ggmlc on Vulkan	95.5%	24 ms
ggmlc on HRX	95.5%	15-16 ms (25 ms at 128 tokens)

Through 1bit serve --laya that is about 28 ms per request at the median. Routing no longer costs a noticeable first-turn delay.

4-bit activations reach the MoE experts#

The Hadamard route rotates a model's weights so 4-bit activations lose less. It now covers the expert matmuls too (#199, #201). Qwen3-Coder-30B-A3B, ROCm, rotated Q4_0:

Route	Prompt (pp512)	Decode	KLD
exact 8-bit activations	1,218	79.0	0.048
4-bit activations, attention and experts	2,156	79.2	0.081

Prompt speed only matters when prompts are long, so tools/e2e_bench.py measures what a user waits for: time to first token plus decode, as effective tokens per second (#202). 1bit serve --long-model sends conversations with 2K or more prompt tokens to the rotated file and shorter ones to the default device. On Qwen3-Coder-30B-A3B, 256 tokens out:

Prompt tokens	Vulkan alone	`--long-model`
512	70.7	71.3
2,048	52.1	53.1
8,192	20.3	25.5
16,384	8.6	13.1

Running rotated files in 1,024-token micro-batches added another 6-10% to prompt speed for free (#205).

GGUFs on the NPU: correct first#

1bit serve -m <gguf> --device npu repacks a GGUF for the NPU and runs it on the per-op forward (#209). Six architectures answer " Paris": MiniCPM5-1B, Qwen2.5-7B, Qwen3-Coder-30B-A3B, Qwen3.6-35B-A3B (GDN + MoE), MiniCPM4-8B (LongRoPE and its embedding and residual scales) and GLM-4.7-Flash (MLA). Getting there meant matching llama.cpp's conventions exactly: rotary pairing, MiniCPM's scales, and the layout of GLM's MLA weights. It is not fast yet: 0.006-0.17 tok/s, because the host sequences every operation. The next step is the fused path, which the NPU work already has for Qwen3.6-35B-A3B (16.3-16.5 tok/s decode through the private add-on).

DwarfStar reads 1BP packages#

DwarfStar now comes from our fork, which reads 1BP packages: a GGUF's tensors and metadata, aligned for streaming (#196). gguf_to_1bp.py converts a local GGUF or one on Hugging Face through range requests, and 1bit serve recognises a .1bp by its first bytes. DeepSeek V4 Flash Q2, converted straight from Hugging Face, gives the same answers and token counts as its GGUF.

HRX so far#

The gates in RFC #213 are measured, not assumed. On ZAYA1-8B, Vulkan decodes at 93 tok/s and HRX at 47.9. On long prompts, HRX already prefills faster than Vulkan: handing an 8K prompt's prefill to HRX cuts a Qwen2.5-7B request by 26%. The decode-split race that made HRX unreliable is fixed (#123), and its multi-pass output step got 15-22% faster on long contexts (#180). Each gate that closes gets its own post.

The review came from geramyL, a moderator on Lemonade's Discord. Thank you. ↩

Qwen3.8-27B at 45.7 tok/s: DFlash2 in 1bit serve, and 4-bit activations with a Hadamard rotation

2026-09-28T00:00:00Z

Qwen3.8-27B is a dense 27B model, and on Strix Halo its decode speed is set by memory bandwidth: every token reads the whole 15 GB file, so plain decode sits near 11-12 tok/s. Prompt processing is set by the GPU's matrix units instead. This week we pushed on both. Every number below was measured on Strix Halo (Ryzen AI Max+ 395, Radeon 8060S), with the method in the linked docs.

Decode: DFlash2 drafts a whole block#

Speculative decoding breaks the bandwidth limit by guessing several tokens and checking them all in one pass of the big model. 1bit serve --mtp already does this with the model's own MTP head. DFlash2, from z-lab and Inco, is a separate 2B draft model trained for Qwen3.8-27B. It emits a whole block of 8 tokens in one pass, reading the big model's hidden states, and keeps the top 16 candidates at each position so it can pick a coherent path through them. Upstream llama.cpp added it as draft-dflash, and the Vulkan build the engine pins already carries it.

The new 1bit serve --dflash <draft.gguf> wires it in. One detail mattered more than any other: llama-server's default draft length is 3, and DFlash only pays when it drafts full blocks. At length 3 it matched MTP; at full blocks it pulled well ahead. So --dflash now drafts full blocks unless you say otherwise.

Qwen3.8-27B Q4_0 through 1bit serve --device vulkan, decode tok/s on three chat prompts (code / prose / short), back to back:

Drafter	tok/s
none	11.1 / 11.8 / 12.4
`--mtp`	31.7 / 25.8 / 28.2
`--dflash` (DFlash2, Q8_0)	45.7 / 28.3 / 17.8

That is 4.1x on code, and 44% over MTP. On prose DFlash2 edges ahead. On the short prompt, a translation that ends after about 16 tokens, MTP still wins: there are too few tokens to fill blocks. The drafter runs faster in Q8_0 than in BF16 (42.5 against 38.9 tok/s on code). A drafted token is kept only when the model itself agrees, so the output is the model's own. Converting the drafter needs only the target's config and tokenizer, not its weights (1bit serve).

Prompt processing: 4-bit activations, rotated first#

On the exact int8 path, prompt processing for this model tops out near 400 tok/s. The GPU is not power-limited: it runs at about 158 W and 2.7 GHz through the whole prompt, and the matmul kernel reaches about half of the int8 matrix rate. The matrix units run int4 × int4 at twice the int8 rate, though, and ROCmFPX's W4A4 kernel uses exactly that instruction.

We pointed that kernel at Unsloth's plain Q4_0 file. A Q4_0 weight is already a 4-bit value with an offset of 8, and flipping its top bit gives the same value in two's complement. So the weights stay exactly what Unsloth shipped, and only the activations are rounded to 4 bits. That is fast (461-488 tok/s) but lossy: mean KLD against the full model rose from 0.029 to 0.084. Leaving any single group of tensors at int8 did not fix it, because the error comes from all of them.

A Hadamard rotation fixes much of it. A few large activation values in a block of 32 force a coarse 4-bit scale on the other 31. Rotating each block by a 32-point Walsh-Hadamard transform spreads that energy evenly before rounding. We rotate the weights the same way once, offline, so the product is unchanged: on the exact int8 path the rotated file matches the plain one (KLD 0.031 against 0.029). The rotation itself runs inside the activation quantizer, as three lane shuffles and two in-register butterfly steps.

Qwen3.8-27B Q4_0 on ROCm	pp512	Mean KLD
exact int8	~400	0.029
W4A4	461-488	0.084
W4A4, Hadamard-rotated	503	0.055

The rotation removes a third of the 4-bit error. Perplexity ends up 1.2% above the exact path, and the error is still nearly double. So this is measured, not shipped: the patches are not in the engine's pin yet (Lean (ROCmFP4, ROCmI4)).

What's next#

DFlash2 is in 1bit serve today. For prompts, the next steps are a better exact int8 matmul for this GPU (about 15% would reach the W4A4 numbers without the error), and rotations larger than 32, which spread outliers further. Packages ship on Sunday with the weekly release.

Update, later on 2026-09-28: the Hadamard route has shipped. tools/hadamard_q4_0.py makes the rotated file, 1bit serve recognises it and runs it on the lean ROCm build by itself, and the Qwen3.8-27B file is published as 1bit-MONSTER/Qwen3.8-27B-Q4_0-H32-GGUF: pp512 509, KLD 0.055, and 440-470 tok/s on a 1,838-token prompt through serve (Lean (ROCmFP4, ROCmI4)).

Laya routing: ask the model what it knows

2026-09-28T00:00:00Z

The engine can run the same model on several devices of one machine: Vulkan and HRX on the Radeon, ROCm, ZINC, and the NPU for its own model format. Someone has to pick one for each request. Step 4 of the port gave that job to Laya, a small decision model that answers typed questions about a piece of text in one forward pass, with a calibrated confidence. This week we wired it into 1bit serve, and the first thing we learned was which question to ask it.

Asking for a device#

The first version asked Laya directly: "which device should run this request?", with the device names as the answers. On eight varied requests (code, a greeting, a translation, a contract summary, an explanation, arithmetic, a refactor, a haiku), it picked the same device every time:

Candidates	Picks
`vulkan,hrx,zinc`	`zinc` 8 of 8
`npu,hrx,vulkan,zinc`	`npu` 4, `zinc` 4

That is not a flaw in Laya. It knows a great deal about text and nothing about this machine, so it had no basis for the answer. We wrote an RFC to change the question, and it was accepted.

Asking about the request#

Now Laya answers a question it can answer: what kind of request is this? Code, prose (an explanation, story or plan), a short answer, or work on a long pasted document. A separate policy file, filled only from measured results, maps each class to a device.

Long documents turned out to be a question of size more than meaning. Laya often read a long pasted log or contract as code, and those were also its slowest decisions, because they fill its whole 512-token window. So any request of 1,024 characters or more is a long document without asking the model, and Laya chooses among the other three.

We measured it on 200 hand-labelled requests, 50 per class:

Setup	Accuracy	code	prose	short	long doc	per decision
typed-decisions checkpoint, size gate	95.5%	100%	82%	100%	100%	588 ms
root checkpoint, size gate	92.0%	94%	74%	100%	100%	401 ms
typed-decisions, four-way question	80.5%	100%	62%	90%	70%	1,739 ms
root, four-way question	64.0%	98%	30%	96%	32%	1,379 ms

The size gate took accuracy from 80.5% to 95.5% and cut the decision time by two thirds. Prose is still the class Laya misses most, reading some of it as a short answer. The confidence is useful: the least confident quarter of the answers is right 84% of the time, the rest 98-100%. Below a confidence of 0.16 a request takes the default device. A test now fails if accuracy drops under 90%.

What the policy says today#

Every class goes to Vulkan. That is a measured result, not a placeholder. Vulkan decodes fastest on Strix Halo for every model file we have measured. The one per-class difference we found is the drafter for speculative decoding. On Qwen3.8-27B, short replies decode faster with the MTP head, and code much faster with DFlash2:

Class	DFlash2	MTP
code	45.7 tok/s	31.7
prose	28.3	25.8
short	17.8	28.2

But llama-server fixes the drafter when it starts. We checked whether a request can change it or switch it off, and it cannot: speculative.type, speculative.n_max and speculative.p_min in a request leave the number of drafted tokens unchanged. Sending short replies to MTP would take a second server holding a second copy of the 27B model, about 16 GB, to save about 0.3 s on a 16-token reply. The policy moves a class only when it has a net gain, so that row stays on Vulkan for now.

What it costs#

Laya decides once per conversation, from its opening messages, and later turns reuse the answer. Measured through 1bit serve --device auto, Qwen3.8-27B with DFlash2:

	1-token reply, first turn	later turn	decode code / prose / short
without Laya	498-518 ms	713-849 ms	50.5-50.7 / 26.7-26.9 / 19.0-19.2
with `--laya`	1,008-1,048 ms	703-767 ms	50.6 / 26.7 / 18.9-19.1

The first turn pays about half a second. After that, nothing changes. Routing is therefore opt-in: --device auto still means Vulkan unless you pass --laya.

Try it#

scripts/fetch-laya.sh                                    # the pinned checkpoint
1bit serve -m model.gguf --device auto --laya
1bit route --laya-model ~/.local/share/1bit/laya --classify --state "Write a Rust function that sums a slice."
# code 0.44 vulkan

Every reply carries an X-1bit-Route header with the class, the confidence and the device, and the server logs each decision. That traffic is what the next policy rows will come from, and the details are in Laya router.

509 tokens a second of prompt: the Hadamard route ships

2026-09-28T00:00:00Z

Earlier today we measured a way to make Qwen3.8-27B read prompts faster on Strix Halo: 4-bit activations on the GPU's matrix units, with a Hadamard rotation to keep them accurate. It is no longer an experiment. The engine now has a tool that makes the rotated file, 1bit serve recognises the file and runs it on the right build by itself, and the Qwen3.8-27B file is on Hugging Face as 1bit-MONSTER/Qwen3.8-27B-Q4_0-H32-GGUF.

Why prompts are a matrix-unit problem#

Decoding reads the whole model once per token, so memory bandwidth sets its speed. Reading a prompt is different: hundreds of tokens go through each weight at once, and the matrix units set the pace. On the Radeon 8060S those units multiply 8-bit integers at one rate and 4-bit integers at twice that rate. The usual path turns activations into 8-bit numbers. On this model it reads prompts at about 400 tokens a second.

A Q4_0 file already stores its weights as 4-bit numbers. A code q means q - 8, and flipping its top bit gives exactly that value in two's complement. So the 4-bit kernel can read Unsloth's Q4_0 weights bit for bit, and only the activations have to go down to 4 bits. That runs at 461-488 tokens a second, but it costs accuracy: KLD against the full model rises from 0.029 to 0.084.

The rotation#

Activations are rounded in blocks of 32 values that share one scale. A few large values in a block force a coarse scale on the other 31, and most of the 4-bit error comes from there. A 32-point Walsh-Hadamard transform spreads each block's energy evenly before rounding, so the scale fits all 32 values better.

The same transform is applied to the weights once, when the file is made. Because the transform is orthogonal, rotating both sides leaves every dot product unchanged: x . w = (Hx) . (Hw). The check is the exact 8-bit path: the rotated file gives KLD 0.031 there, against 0.029 for the same model unrotated. On the GPU the rotation runs inside the step that quantizes activations: two butterfly steps within each thread and three lane shuffles.

The numbers#

Qwen3.8-27B on Strix Halo, KLD against BF16 over wikitext-2, prompt speed from llama-bench (512 tokens):

	Prompt, tok/s	Perplexity	KLD	Same top token
Q4_0, 8-bit activations	~400	6.021	0.029	91.9%
Q4_0, 4-bit activations	461-488	6.241	0.084	87.4%
Q4_0-H32, rotated, 4-bit activations	509	6.096	0.055	89.3%

The rotation removes a third of the 4-bit error. Against the 8-bit path, the rotated file reads prompts 28% faster for 1.2% more perplexity. Through 1bit serve, a 1,838-token prompt runs at 440-470 tokens a second.

Making the file is part of the engine#

A rotated file only works when the runtime rotates the activations too. Run it anywhere else and it produces garbage. So the engine treats the file and the route as one thing.

tools/hadamard_q4_0.py rotates the attention, FFN and delta-net projections of a Q8_0 or BF16 source, adjusts the imatrix to match, and quantizes. It then stamps the file and checks that every Q4_0 tensor in it is a rotated one; a file that fails is deleted. Qwen3.8-27B took about 20 minutes.
1bit serve reads the stamp. With no device flag it runs the file on the lean ROCm build with the rotation and the 4-bit path turned on. It refuses the file on any other device.
A test covers the routing without a GPU, so CI checks it on every change.

tools/hadamard_q4_0.py Qwen3.8-27B-Q8_0.gguf Qwen3.8-27B-Q4_0-H32.gguf --imatrix imatrix.gguf
1bit serve -m Qwen3.8-27B-Q4_0-H32.gguf

Two more kernels on the way#

Qwen3.8's linear-attention layers had two slow steps on ROCm, and the same update fixes both. A new prefill kernel keeps each state column in registers and stages each token's inputs in shared memory. A tiled transpose replaces the strided copy that fed their convolution. Both match the CPU reference in the backend tests, and every model with these layers gets them on ROCm.

What it is for#

This route reads prompts. It decodes at about 11 tokens a second, so long answers belong elsewhere: the Vulkan route with the DFlash2 drafter decodes code at 45.7 tokens a second on the same model. Use the rotated file where prompts are long and answers short, like summaries, code review over whole files, or retrieval with large contexts. The details are in Lean (ROCmFP4, ROCmI4).

Milestone: the HRX decode race is solved

2026-09-27T00:00:00Z

Yesterday's post ended with a workaround. HRX's split flash-attention decode kernel gave different answers to identical requests, and on Qwen3-Coder-30B it sometimes produced NaNs that faulted the GPU (#123, #140). So 1bit serve --device hrx stopped using it. Today we found the cause and fixed it, and the kernel is back on by default. It is correct again, and faster than the path we used without it.

What it was#

The decode-split kernel attends over the KV cache in blocks, and a final workgroup combines the blocks. That workgroup writes the attention output to global memory. Other waves then read it back to quantize it (next_q8) for the next projection. Between the write and the read sat a workgroup barrier that fences shared memory (LDS) only. It does not wait for global stores or invalidate the cache the readers use. So a reading wave could pick up the output slot's previous contents. The f32 output was right, the quantized copy was wrong, and when the stale bytes happened to decode as NaN the MoE router saw NaN logits.

The window is narrow on a quiet box (a few percent of requests diverged there) and wide when the GPU preempts the dispatch in the middle. Any process's KFD queue eviction does that: page compaction, KSM, memory pressure. That is why the failure rate followed those system settings. For a while it looked like a page-migration bug, but migration was only the trigger. The fix fences both memory spaces at all five places the kernel does this: barrier<global>, then barrier<workgroup> (HRX).

How we know#

A separate process forced queue evictions while the engine served, to make the rare case common:

Qwen3-Coder-30B-A3B (2113-token prompt): 1 of 8 correct before, 8 of 8 after, with no NaN.
Qwen3-0.6B, greedy: 13-34 divergent answers per ~177 requests before, none in 354 after.

The last clue came from hashing every kernel's output in serial execution. In every bad request, the first command whose output diverged was this kernel, with the f32 output identical and the quantized output different.

Faster with it on#

Decode tok/s (Q4_K_M, llama-bench, interleaved on a shared box):

model	ctx 2100, split on	ctx 2100, split off
Qwen3-0.6B	140-158	118-122
Qwen3-Coder-30B-A3B	42-62	47-49
ZAYA1-8B	38-42	34-36

The kernel's multi-pass output step was also vectorised (#180): on Qwen3-Coder-30B-A3B, +15% at depth 2100, +20% at 3000 and +22% at 4800 (median of 6 interleaved rounds), with bit-identical perplexity. ONEBIT_HRX_DECODE_SPLIT=0 turns the kernel off if you need to compare.

Also this week#

Architecture gaps closed. OPT, GPT-Neo, CodeGen and GPT-J, which upstream llama.cpp has no model code for, and Zyphra's whole family, from ZAYA1-74B to the vision models, all run from our llama.cpp. The registry now maps 323 Hugging Face architectures, up from 265 at step 5 (Model registry).
GGUF on the NPU: a repacked GGUF serves on the NPU. Qwen2.5-7B, MiniCPM4-8B and MiniCPM5-1B answer there (#179, NPU).

Packages with the fix ship on Sunday with the weekly release.

Zyphra's whole family, four older architectures, and experts from the drive

2026-09-26T00:00:00Z

A lot landed this week. Zyphra's model family now runs end to end, including its vision models. Four older architectures that upstream llama.cpp never shipped are in. 1bit serve can run a MoE model whose experts do not fit in memory. And the HRX decode path became deterministic. Every number below was measured on Strix Halo (Ryzen AI Max+ 395, Radeon 8060S), with the method in the linked docs.

Zyphra, all of it#

ZAYA1-8B already ran on Vulkan, HRX and ROCm. The rest of the family followed:

ZAYA1-74B-preview, with its sliding-window layers: Q4_K_M decodes at 35.4 tok/s on Vulkan.
ZAYA1-base and ZAYA1-reasoning-base, in Zyphra's older checkpoint layout. Our converter reads that layout directly, and the tensors it writes are byte-identical to ZAYA1-8B's.
ZAYA1-VL-8B, the vision model. It applies a vision-only LoRA to image tokens and attends to each image bidirectionally. Against Zyphra's own FP32 code it agrees at 100 of 101 teacher-forced positions. F16 decodes at 38-51 tok/s. It reads "HELLO 42" off a synthetic image, and it recognises the New York Times front page of the moon landing.
Zamba, Zamba2 and BlackMamba, plus Zamba2-VL. Zamba v1 and BlackMamba match the reference at 96/96 positions. A new Vulkan scan for Mamba-1 layers took Mamba-370M from 25.8 to 173.5 tok/s; those layers used to run on the CPU.

Adding the vision model left ZAYA1-8B's text path unchanged. Its wikitext perplexity is identical on Vulkan (21.5731, and 21.6213 with four sequences per batch) and HRX (21.5518), and it still agrees with transformers at 95/96. Serve the vision models with 1bit serve --mmproj (Vulkan (upstream)).

Four older architectures#

OPT, GPT-Neo, CodeGen and GPT-J now run from our llama.cpp. We checked each port against transformers FP32 before it merged. CodeGen started at 0/96: its fused projection stores query, value and key in that order, not query, key, value. After the fix all four agree at every checked position (Vulkan (upstream)). With them, the model registry maps 323 Hugging Face architectures to a backend. "Mapped" means a backend's code accepts the architecture; the census reports separately how many have been checked (Model registry).

MoE experts from the drive#

1bit serve --moe-slots N keeps a MoE model's routed experts in the file on disk. It holds N of them in GPU memory and streams in the rest as the router asks for them. Perplexity is exactly that of the all-GPU model. On Qwen3-Coder-30B-A3B, with 75% of the experts resident and the prefetch moved off the decode path, it decodes at 43-45 tok/s (39 through serve). With 25% resident it still decodes at 8-10. The resident model is faster (79-91 tok/s), so streaming is for models that do not fit (MoE streaming).

HRX decode, now deterministic#

Repeating the same request exposed a problem in HRX's split flash-attention decode kernel: it gave different answers to identical requests, and on Qwen3-Coder-30B it faulted the GPU in two of three runs. 1bit serve --device hrx now decodes without it by default. That path is deterministic, and on most models faster: Qwen3-0.6B goes from 169 to 322 tok/s, and ZAYA1-8B from 23.5 to 25.5. Coder-30B runs 5 of 5 clean at 66-71 tok/s. ONEBIT_HRX_DECODE_SPLIT=1 turns the kernel back on for testing a fix (HRX).

Also#

ONNX on the GPU on Linux: ONNX Runtime's WebGPU provider, over Vulkan. Qwen3-4B int4 decodes at 52.9 tok/s on the Radeon, against 26.8 on the CPU (ONNX Runtime).
GPU matrix units: a short reference on the Radeon's matrix instructions. int8 runs at the f16 rate; only int4 × int4 doubles it (HRX).
Issues and security:
- issue forms, and RFCs for larger changes in Discussions;
- a CI check that stops a pin from moving backwards;
- CodeQL's extended suite, with no open alerts.

Packages for all of this ship on Sunday with the weekly release.

The NPU drafts, the GPU checks

2026-09-25T00:00:00Z

Speculative decoding makes a large model faster by letting a small one guess ahead: the small model proposes a few tokens, the large model checks them all in one pass, and every guess it agrees with is a token it did not have to produce one at a time. Usually both models run on the GPU. On Strix Halo there is a second processor sitting idle during chat: the XDNA 2 NPU. So we gave it the small model.

How it works#

The NPU runs Qwen3-0.6B on the engine's fast lane (91 tokens per second on its own). The GPU runs the target, a Qwen3 model from 14B to 32B, through llama.cpp on Vulkan. Each round:

The NPU proposes k tokens, one after another.
The GPU scores all k in a single batch and predicts the token after each.
The longest run of proposals the GPU agrees with is kept, plus the GPU's own next token.
Both key/value caches roll back to that point, and the next round starts.

Both models decode greedily, so the result should equal the GPU decoding alone, token for token. That is the correctness check we ran every time.

One constraint: the two models must share a tokenizer. Qwen3-0.6B can draft for the Qwen3 family (151,936 tokens), including Qwen3-14B and Qwen3-32B; the newer Qwen3.5, 3.6 and 3.8 models use a different vocabulary.

Measured#

Unsloth's UD-Q4_K_XL files on Strix Halo, a coding prompt, 192 tokens (136 for the 14B, which finished its answer):

Target	k	GPU alone	NPU drafts, GPU checks	Speed-up	Drafts accepted
Qwen3-32B	2	11.0 tok/s	17.9 tok/s	1.63x	72%
Qwen3-32B	4	10.2	21.3	2.09x	59%
Qwen3-32B	6	10.7	23.4	2.19x	56%
Qwen3-32B	8	10.7	12.9	1.20x	53%
Qwen3-14B	2	24.1	31.6	1.31x	71%

The larger the target, the more it pays. Qwen3-32B more than doubles; the faster 14B gains 31%.
k = 8 is past the knee. Checking nine tokens at once costs the GPU far more than checking seven.
The output matched GPU-only decoding in six of eight runs. The other two differed at a single token, where the model's top two choices were nearly tied and the GPU's batched arithmetic broke the tie the other way.

Against drafting on the GPU#

The same 0.6B drafter can also run on the GPU next to the target, which llama.cpp does built in. We measured that too, same prompt and files:

Target	GPU alone	0.6B drafts on the GPU	0.6B drafts on the NPU
Qwen3-32B	10.4 tok/s	24.9 tok/s	23.4 tok/s
Qwen3-14B	22.5	43.5	31.6

Drafting on the GPU is still a little faster: a 0.6B model costs the GPU very little, and in this first version the NPU and the GPU take turns. What the NPU gives is the drafting done on silicon that was otherwise idle, with the GPU never running the small model at all. Making each NPU draft step cheaper is the next piece of work.

The benchmark is tests/npu_spec_bench.cpp on the npu-draft branch.

Speculative decoding: where it pays, and where it does not

2026-09-24T00:00:00Z

Speculative decoding lets a cheap drafter guess the next few tokens and the model check them all in one pass. When the guesses are right, several tokens come out of one pass instead of one. We measured it on the two most popular Unsloth models on Strix Halo, and the answer depends entirely on the model.

Dense Qwen3.8-27B: up to 3.4x#

Qwen3.8-27B ships its own multi-token-prediction (MTP) head, a small extra layer trained to draft. 1bit serve --mtp switches it on. Decode speed on three chat prompts (code, prose, short), UD-Q4_K_XL on Vulkan:

Setting	Code	Prose	Short
No drafting	12.2	12.3	12.3
`--mtp` (draft 3, the default)	35.9	28.2	28.8
`--mtp --mtp-max 6 --mtp-p-min 0.5`	42.0	27.3	25.2
`--mtp --mtp-max 8`	21-25	13-20	10-20

The default is the all-rounder: 2.3-2.9x on every prompt. Code is predictable enough to keep long drafts (93% of drafted tokens accepted), so draft length 6 with a confidence threshold of 0.5 reaches 3.4x there. Prose is not (56-76% accepted), and at length 8 the rejected drafts cost more than the accepted ones save. Adding free n-gram drafting on top of MTP gained nothing.

Accuracy does not change: a drafted token is kept only when the model would have produced it anyway.

Qwen3-Coder-30B-A3B: never#

Qwen3-Coder-30B-A3B is a mixture of experts with 3B parameters active per token. It already decodes at 88 tok/s on Vulkan, and it has no MTP head, so we tried separate draft models (Qwen3 0.6B, 1.7B and 4B, 36 combinations of draft length and threshold on Vulkan and ROCm) and three n-gram methods:

Method	tok/s on code
No drafting	88
n-gram drafting	89 (a tie)
Qwen3-0.6B draft, length 4	84, with 82-87% of drafts accepted
Qwen3-1.7B draft	41-66
Qwen3-4B draft	20-44

Even a draft model that is right five times out of six loses: checking its guesses costs almost as much as generating them, when the model only reads 3B parameters per token.

The rule#

Speculative decoding pays when the model is expensive per token and brings its own drafter: dense models with an MTP head. It does not pay on small-active mixtures of experts. And it is a single-user lever: a server with MTP loaded batches about a third less well, so under load the engine keeps MTP and batching apart (see 1bit serve).

Smaller files: Unsloth, ROCmFPX and a ternary 27B

2026-09-24T00:00:00Z

A smaller model file decodes faster, because decode is limited by how many bytes the GPU reads per token. The question is what each format costs in accuracy. We measured three families of small files on Strix Halo against their full-precision originals: KL divergence (how far the model's predictions move) and how often the top predicted token stays the same.

Qwen3.8-27B: Unsloth against ROCmFPX#

Every file below is quantized from the same BF16 checkpoint, the ROCmFPX ones with Unsloth's own importance matrix so the comparison is fair.

File	Size	KL divergence	Same top token	Decode (Vulkan)
UD-Q4_K_XL (Unsloth, the default)	16.4 GiB	0.008	95.3%	11.9 tok/s
UD-IQ4_XS (Unsloth)	13.3 GiB	0.019	93.3%	15.0
UD-Q3_K_XL (Unsloth)	12.2 GiB	0.028	92.1%	15.8
ROCmI4 (ROCmFPX)	13.9 GiB	0.036	91.3%	13.5 (ROCm)
ROCmFP4 (ROCmFPX)	13.8 GiB	0.045	89.6%	14.1
ROCmFP2 (ROCmFPX)	8.6 GiB	0.341	75.5%	21.2

Unsloth's own UD-Q3_K_XL is smaller, faster and closer to the model than either 4-bit ROCmFPX format. ROCmFPX keeps two jobs in the engine's lean option (docs): ROCmI4 for the fastest prompt processing we measured (455 tok/s with its W4A4 path on ROCm), and ROCmFP2 when memory is the limit.

A model trained ternary#

PrismML's Ternary-Bonsai-2-27B is trained with weights of -1, 0 or +1, not squeezed into them afterwards. Its files are 5.5 GiB (PTQ1_0, 1.75 bits per weight) and 6.7 GiB (PQ2_0), against 50 GiB in F16. Measured against its own F16:

File	Where	KL divergence	Same top token	Decode
PQ2_0	CPU	0.00014	99.6%
PTQ1_0	ROCm	0.065	91.3%	27.3 tok/s

The ternary format itself is effectively lossless: on CPU, PQ2_0 matches F16 to within 0.00014. At 27 tok/s the 27B decodes 2.3x faster than the 4-bit file above, with no drafting. What is not there yet is the GPU path: PrismML's llama.cpp fork has no Vulkan kernels for these formats, its ROCm PTQ1_0 kernel loses some accuracy, and its ROCm PQ2_0 kernel is broken. A 27B at full quality in 7 GB is one kernel fix away.

One trap on the way#

hipBLAS returns wrong matrix products on this GPU (gfx1151; ROCm/rocm-libraries#11530). It never touched our 4-bit files, which use llama.cpp's own kernels, but it silently corrupted every F16 measurement until we moved those to the CPU and to Vulkan. The engine's ROCm build now forces llama.cpp's own kernels everywhere; that also measured slightly more accurate (KL 0.0064 against 0.0094) at the same speed.

One GPU, many users

2026-09-24T00:00:00Z

A chat with one person and a server for many want different things from the same GPU. We measured Qwen3.8-27B (UD-Q4_K_XL) on Strix Halo from 1 to 24 simultaneous requests, and the engine now grows with the load.

Batching is the multi-user lever#

Decode reads every weight once per step. With one request, one token comes out of that read; with eight batched together, eight do. 1bit serve --parallel N gives the model N slots that decode together:

Requests	Vulkan	ROCm
1	11.8	11.6
4	36.9	30.2
8	50.9	35.6
16	42.9	68.0

Total tok/s. Vulkan is the better batcher up to 8 requests and then falls back; ROCm keeps climbing to 16, where it serves 5.8x one stream. On the smaller-active Qwen3-Coder-30B-A3B the same pattern holds at a higher level: 228 tok/s on Vulkan at 8 requests, 319 on ROCm at 16.

Two backends on one GPU also help a little on their own: Vulkan and ROCm decoding at the same time made 14.3 tok/s together against 12.2 for either alone (+18%), two Vulkan processes only +4%. They are one GPU on one memory bus, so it is a small gain, not a doubling.

MTP and batching do not mix#

Multi-token prediction gives one user 2.3-3.4x (the previous post). Under load it costs: with the MTP head loaded, Vulkan at 4 requests made 24.9 tok/s against 36.9 without it, and turning drafting off per request (draft length 0) did not win it back. Having MTP loaded changes how the server batches.

Growing with the load#

So 1bit serve --adaptive runs two backends side by side, the model loaded on each, and routes every request by how busy they are:

Requests	1	2	4	8	12	16	24
`--adaptive --adaptive-at 8`	12.0	21.9	36.3	49.9	43.5	44.7	63.5

Vulkan takes the first eight requests, batched; ROCm takes the overflow. It matches the best single setup up to 8 and keeps going past it: 63.5 tok/s at 24, where Vulkan alone falls back. Between 12 and 16 the two backends compete for the one GPU and it dips below ROCm alone; that is the next thing to tune.

The first version of the router had a bug worth recording: a burst of requests all checked "is Vulkan free?" before any of them had claimed it, and all piled onto one slot. Choosing a backend and reserving its slot is now one locked step.

Which to use#

One person at a time: 1bit serve --mtp (35-42 tok/s on Qwen3.8-27B).
Many people: 1bit serve --adaptive --adaptive-at 8.
A fixed crowd of 16 or more: --device rocm --parallel 16.

Details and the full tables are in 1bit serve.

1bit.MONSTER is now 1bit engine

2026-09-24T00:00:00Z

1bit.MONSTER grew fast: an inference engine, a model zoo, benchmarks, a store, a voice assistant and a lot of experiments, all in one repository with all of its history. The engine that came out of it is now its own project, 1bit engine, and this site is its home.

What changed#

A clean repository. The engine lives in 1bit-MONSTER/engine. It holds verified code only, without the development history. Each piece is ported over once it passes its end-to-end test on Strix Halo, and the porting map shows what has moved and what is next.

Apache-2.0. The old repository was GPL-3.0. The engine is Apache-2.0, with a copyright and license notice on every file, enforced in CI, and every third-party project it builds on credited in its NOTICE.

Inside Lemonade, not around it. The old engine vendored its own copy of Lemonade. Now Lemonade is the server you talk to (its catalog, downloads, router and UI), and it runs the engine as one of its backends, through 1bit serve, the same way it runs llama-server.

Upstream, pinned and kept current. The XDNA driver and XRT, AMD's HRX pair, llama.cpp for Vulkan, ZINC, the Linux kernel, Hugging Face tokenizers and the Laya router are each pinned to upstream, and the pins are bumped automatically, so new models and fixes arrive soon after upstream ships them.

What it runs on#

One 1bit serve drives:

the XDNA 2 NPU, on full ELFs generated in C++;
HRX and Vulkan on the Radeon iGPU, one llama.cpp build with both;
ZINC, which also reaches NVIDIA GPUs through CUDA;
MLX on Apple Silicon.

Measured numbers live on the wiki. The first week of 1bit engine lists what the first three days produced.

The old site#

The 1bit.MONSTER site and its posts stay online in the 1bit.MONSTER archive, and old links land on a page that points at their archived copy. The blog page here lists the old posts worth keeping. 1bit JARVIS, the voice assistant, is not part of the engine for now; it will come back later as its own piece.

Prefill on HRX, decode on Vulkan, one KV cache

2026-09-24T00:00:00Z

On Strix Halo's Radeon iGPU, AMD's HRX runtime prefills a prompt faster than Vulkan, and Vulkan decodes faster than HRX. 1bit serve --device vulkan --prefill-device hrx now uses both on the same request, and the key-value cache never gets copied between them.

The idea#

A request has two phases. Prefill reads the whole prompt at once and is compute-bound; decode makes one token at a time and is bound by memory bandwidth. Our one llama.cpp build carries both GPU backends, HRX0 and Vulkan0, so the engine loads the model on each, runs the prompt prefix on HRX, and hands the rest of the request to Vulkan.

The hand-off is the hard part. Prefill's output is the KV cache: every layer's keys and values for every prompt token. Copying it from one backend to the other would cost the time the split saves.

Zero copies#

So both contexts use one KV cache:

HRX allocates it in its own device memory and exports it as a dma-buf (through HSA).
Vulkan imports the same memory with VK_EXT_external_memory_dma_buf and reads it in place, at 110.6 GB/s against 113.2 GB/s for its own allocations.
The only thing that moves between them is the cache's bookkeeping: which cells hold which positions of which sequence.

We tried sharing host memory first. It worked, but it cost Vulkan half its decode speed; device memory through a dma-buf does not.

Both sides compute the cache's layout independently, so they must agree on it exactly: tensors on 4 KiB boundaries, split into chunks of at most 1 GiB (Vulkan reads garbage from a single buffer past about 4 GiB), and a hash of the layout checked on both sides. HRX's prefill is fastest in whole batches, so only whole ubatches of the prompt go to HRX, and short prompts skip the split: below 1024 tokens (--prefill-min-tokens) it does not pay.

Measured#

llama-server on Strix Halo, prefill and whole request:

Model	Prompt	Vulkan alone	HRX prefill + Vulkan
Qwen2.5-7B Q4_K_M	2048	1541 / 4314 ms	1078 / 3890 ms (-10%)
Qwen2.5-7B Q4_K_M	8192	7347 / 10376 ms	4646 / 7668 ms (-26%)
Qwen3-0.6B Q4_K_M	8192	1289 / 2267 ms	1089 / 2032 ms (-10%)

Decoding on the shared cache runs within 3% of Vulkan with its own cache. The output is the same model: against Vulkan alone, teacher-forced over 64 tokens, the mean KL divergence is 0.0002-0.0006 nats and the top token agrees on 64-65 of 65 positions. For scale, Q4_K_M quantization itself is 0.063 against BF16.

Where it lives#

The patches are in our llama.cpp fork, on top of AMD's tested HRX commit, and are rebased onto every new AMD pair automatically: dma-buf export in ggml-hrx, dma-buf import in ggml-vulkan, the cache sharing in llama, and the prefill hand-off in llama-server. The weights are loaded twice, once per device, so it suits machines with memory to spare; Strix Halo's unified memory is one. The details are in the HRX docs.

HRX answers long prompts correctly, and where Unsloth quants pay

2026-09-24T00:00:00Z

Two results from the end of the day: a bug fixed in HRX decoding, and one table that answers "which Unsloth file should I download?" for five popular models.

HRX: decoding past 256 tokens#

1bit serve --device hrx gave garbled answers to prompts of about 400 to 1,700 tokens, while short prompts and very long ones read fine. Prompt processing was right at every length; what went wrong was decoding one token at a time once the KV cache held more than 256 tokens.

HRX's decode attention splits the cache into 64-token blocks and then combines the per-block results. Up to 256 tokens it combines them one way; past that it switches to a cooperative path in which four groups of threads each take two query rows. That path only checked that a row's query head exists, not that the row belongs to the key/value head doing the work. Qwen3-0.6B has two query heads per key/value head, so the spare groups combined stale rows and wrote them over their neighbours' output. The fix adds that one bound, in both copies of the kernel.

After the fix, fed the same tokens as Vulkan, HRX's predictions differ by 1-3% at every length we tried (248 to 2,040 tokens), and chat answers match Vulkan's from 404 to 2,844 tokens. The details are in the HRX docs.

The same day, the sub-4-bit Unsloth files started working on HRX: our pinned HRX build has an IQ3_XXS matmul, and HRX and Vulkan now give the same perplexity within 0.3% on UD-Q2_K_XL and UD-IQ2_M. They decode slowly there (13-17 tok/s, where Vulkan does about 390 on Qwen3-0.6B) because parts of those files still run on the CPU, so Vulkan remains the device for them.

Which Unsloth file to download#

For each model we measured every Unsloth Dynamic quant from 2 to 5 bits against a reference (BF16 when it fits, else Q8_0): KL divergence over 40 x 512 tokens of wikitext-2, and decode speed on Vulkan on Strix Halo. Two picks come out of it on every model:

Model	Lean pick	KL	Decode	Accurate pick	KL	Decode
Qwen3.5-4B	UD-Q4_K_XL, 2.7 GiB	0.018	60.0 tok/s	UD-Q5_K_XL, 3.0 GiB	0.0099	55.4 tok/s
Qwen3.5-9B	UD-Q4_K_XL, 5.6 GiB	0.016	36.0	UD-Q5_K_XL, 6.3 GiB	0.0092	32.7
Qwen3-Coder-30B-A3B	UD-Q4_K_XL, 16.5 GiB	0.027	84.7	UD-Q5_K_XL, 20.2 GiB	0.012	76.4
Qwen3.6-27B	UD-Q4_K_XL, 16.4 GiB	0.018	12.0	UD-Q5_K_XL, 18.7 GiB	0.0081	10.0
Qwen3.6-35B-A3B	UD-Q4_K_XL, 20.8 GiB	0.014	61.0	UD-Q5_K_XL, 24.8 GiB	0.0091	53.7

UD-Q4_K_XL is the lean pick everywhere: the smallest file that stays within a KL of 0.03.
UD-Q5_K_XL is the accurate pick: under 0.01 on four of the five models, for 8-17% slower decode. Qwen3-Coder-30B-A3B is the exception: 0.012 is as close as any Unsloth file gets it.
Skip the 2-bit files at these sizes: KL 0.11-0.23, with the top prediction changing on 13-21% of tokens.

One popular model is missing: gemma-4-12b-it does not run correctly on the llama.cpp build we measured with, so there was nothing fair to compare. (MiniMax-H3 was on our download list too, by mistake: it generates video, it is not a text model.) Every row, with prompt speeds and ROCm, is on the Quantization page of the wiki.

The first week of 1bit engine

2026-09-24T00:00:00Z

The clean repository started on 2026-09-22. In its first three days, the engine came up on every device it targets, inside Lemonade, with each step verified on real hardware before it merged. These are the milestones, with the numbers and where they come from.

The NPU, on full ELFs generated in C++#

The XDNA 2 NPU now runs without an xclbin and without per-context kernel files. The engine generates each kernel's full ELF (the design plus its control code) in C++ when the model loads, and XRT opens it directly; for each token, the host submits every layer and the lm head as one runlist (#8, #9).

Qwen3-0.6B decodes at 91 tok/s (11.0 ms per token), and 87 tok/s served through Lemonade.
The XDNA driver and its XRT are pinned to upstream and built privately (#12).
The layer kernel itself is being rebuilt from scratch as our own open kernel ("dx", on IRON and Peano). RMSNorm, RoPE and attention pass their tests so far, correctly rounded to bf16.

HRX and Vulkan in one build, then both at once#

The Radeon iGPU has two devices in one llama.cpp build: HRX0 on AMD's HRX runtime and Vulkan0. HRX follows AMD's live ggml-hrx, pinned to the pair AMD tests and bumped daily (#11); Vulkan has its own pin on upstream llama.cpp's latest release (#28).

HRX went from 3-5x slower than Vulkan to close behind it. Its prefill is now 1.3-1.7x Vulkan's, and its decode within 10% on 4-bit files.
Our patches make HRX correct on more models (#29): an IQ3_XXS kernel, and op claims that only take the nodes HRX can actually run. test-backend-ops on HRX0 went from about 1150 failures to 791 passing, 0 failing, and Unsloth's sub-4-bit files, which failed on HRX, now come within 1% of Vulkan's perplexity.
Prefill on HRX, decode on Vulkan, with zero copies (#30): a whole 8192-token request on Qwen2.5-7B finishes 26% sooner. How it works.

Beyond AMD#

NVIDIA, through ZINC's CUDA backend (#14): Qwen3.5-9B at 167-173 tok/s on an RTX 5090, and 295 tok/s through ZINC's Vulkan backend on Strix Halo.
Apple Silicon, through Lemonade's mlx recipe and lemon-mlx-engine, verified on an M4 (#10).

Underneath#

Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI reads any model's tokenizer.json, byte-exact on 18 models (#19).
Quantization: on the GPU, Unsloth's Dynamic UD-Q4_K_XL beats Q4_K_M: 17% lower KL divergence for 2% more size. On the NPU, a ternary model converts to 4-bit exactly, and XDNA 2 multiplies int8 by int4 natively (wiki).
The Linux kernel is pinned to upstream v7.3-rc4 with amdxdna in-tree (#17), and the Laya router's source and checkpoints are pinned and hash-verified (#18).

Next#

Step 4 is the Laya router, which decides where each request runs. After it: every Hugging Face architecture, kept current by a daily census. The porting map tracks it, and the wiki keeps every measured number with how it was taken.