Vulkan from upstream llama.cpp#

1bit serve --device vulkan runs a llama-server built from upstream llama.cpp (MIT), pinned to its latest release in third_party/llama.cpp-vulkan, plus the few upstream pull requests we carry until upstream merges them ("Carried upstream PRs" below). It is a separate tree from third_party/llama.cpp, which stays on the llama.cpp + hrx-system pair AMD tests together for HRX (HRX).

Two pins, because the two routes stay current in different ways:

A new model architecture reaches the Vulkan route the day upstream releases it, without waiting for AMD's pair to move. Qwen3.8-Flash-Next (qwen4exp) is the case that prompted this: upstream added it on 2026-08-27, and AMD's pinned llama.cpp (f1a0aca141de) cannot load it.

Carried upstream PRs#

1bit/vulkan-upstream is ggml-org's release plus these commits. On each new release bump-llama-vulkan.yml rebases them onto it; one that upstream has merged becomes empty and drops out, and one that no longer applies stops the bump for a hand rebase. The commit the engine pinned before is kept as the tag vulkan-upstream-<sha12>.

Each upstream PR is reviewed line by line before it is pinned. #28243: it touches only the qwen4exp model, its converter, and two lines of the shared speculative decoding (the -md path fix, and KV sharing kept to gemma4-assistant drafts, as v0.5.0 already did). A draft head can only change speed: the target model checks every drafted token.

Build#

Needs the Vulkan headers, loader and glslc.

git submodule update --init --depth 1 third_party/llama.cpp-vulkan
cmake -B build -G Ninja -DONEBIT_VULKAN=ON
cmake --build build

ONEBIT_VULKAN and ONEBIT_HRX build side by side: --device vulkan then uses the upstream build and --device hrx the HRX build. Without ONEBIT_VULKAN, --device vulkan falls back to the HRX build's Vulkan backend, as before. ONEBIT_LLAMA_SERVER still overrides both.

Verified (Strix Halo)#

v0.5.0 7fe450e19305:

v0.5.0 + ggml-org#28243 (62484fba), llama-server built from the branch:

v0.5.0 + #28243 + #21412 + the d_state-64 scan (a32f0d6f); Zamba2 converted with the carried converter (convert_hf_to_gguf.py --outtype q8_0), test-backend-ops SSM_SCAN 14/14 on Vulkan0:

Mamba-1 scan (adb2af5d), state-spaces/mamba-370m-hf F16, llama-bench -p 256 -n 64: pp256 228 → 6736 tok/s, tg64 25.8 → 173.5 tok/s; greedy completions identical to the CPU-scan build; test-backend-ops SSM_SCAN 17/17.

Zamba v1 (ee6e4e0d), Zyphra/Zamba-7B-v1 Q8_0: teacher-forced top-1 vs transformers FP32 96/96 on three prompts; greedy continuations identical to the reference's; pp256 543, tg64 14.9 tok/s; 2 graph splits (all compute on Vulkan0: each layer's two-head scan runs on the Mamba-1 Vulkan kernel from adb2af5d). GGUFs of Zamba or Zamba2 converted before ee6e4e0d have the broken token scores: reconvert them.

BlackMamba (f6261222), Zyphra/BlackMamba-1.5B: teacher-forced top-1 vs Zyphra's own PyTorch code (FP32, CPU) 96/96 on three prompts, identical greedy text and prompt tokens; F16 pp256 7180 / tg64 250 tok/s, Q8_0 pp256 10136 / tg64 369 tok/s. No HF architectures in its config.json: the converter routes on mamba_moe_layers.

Zamba2-VL (f8132a7f): images through 1bit serve --mmproj (1bit serve).

python third_party/llama.cpp-vulkan/convert_hf_to_gguf.py --remote Zyphra/Zamba2-VL-2.7B --outtype q8_0 --outfile zamba2-vl-2.7b-Q8_0.gguf
python third_party/llama.cpp-vulkan/convert_hf_to_gguf.py --remote Zyphra/Zamba2-VL-2.7B --mmproj --outtype f16 --outfile mmproj-zamba2-vl-2.7b-F16.gguf
1bit serve -m zamba2-vl-2.7b-Q8_0.gguf --mmproj mmproj-zamba2-vl-2.7b-F16.gguf --device vulkan --ctx-size 8192

The vision embeddings were checked against transformers' Qwen2.5-VL vision tower in FP32, loaded with Zamba2-VL-2.7B's weights and fed by its own image processor: 192 tokens × 2560, mean per-token cosine 0.99989 on the CPU and 0.99944 on Vulkan0. The test image is a red square, a blue circle and the text "HELLO 42":

The 1.2B and 2.7B misread the text the same way on the CPU and on Vulkan, so the misreading comes from the models, not from the port.

GGUFs made before llama.cpp #17 carry a chat template that left each image inside the user turn instead of in front of it, as Zyphra's template has it. llama-server marks images with <__media_<id>__>, an id random per server, and that template knew only <__media__>. Reconvert those GGUFs, or serve them with --chat-template-file. With the new template the image starts at the same prompt position as in Zyphra's processor.

Large models need --ctx-size: without it llama-server allocates the KV cache for the model's full trained context (262,144 tokens for Qwen3.8), which does not fit next to Flash-Next's 104 GiB of weights.

Measured quant sweet spots for Qwen3.8 on this route are on the wiki.

ZAYA1 (Zyphra), from our llama.cpp#

ZAYA1 is Zyphra's MoE: 8.8B parameters, 760M active per token. Every layer runs CCA attention, where q and k pass through short convolutions over time, then a top-1 router that picks one of 16 experts or a skip expert. Upstream llama.cpp has no ZAYA: its draft (ggml-org/llama.cpp#23112) was closed on 2026-09-05. ZAYA lives in our llama.cpp, third_party/llama.cpp (1bit-MONSTER/llama.cpp, PR #2), and runs there on Vulkan, HRX and ROCm (HIP). 1bit serve --device vulkan reads the GGUF's general.architecture, and runs an architecture that only our llama.cpp has on that build's Vulkan0 even when the upstream build is present; --device hrx runs it on HRX0.

python third_party/llama.cpp/convert_hf_to_gguf.py <Zyphra/ZAYA1-8B> --outtype f16 --outfile zaya1-8b-f16.gguf
build/hrx/llama/bin/llama-quantize zaya1-8b-f16.gguf zaya1-8b-Q4_K_M.gguf Q4_K_M
1bit serve -m zaya1-8b-Q4_K_M.gguf --device vulkan --ctx-size 8192

ZAYA GGUFs converted before 2026-09-27 tokenize chat turns wrongly. Their <|im_start|> (id 105) and <eos> were stored as ordinary tokens, so llama.cpp spelled each turn marker out as seven text tokens and chat answers went off-template ("12 + 30" gave "22"). The converter marks them as control tokens since llama.cpp #27, and every ZAYA GGUF on the 1bit-MONSTER Hugging Face org was fixed in place the same day (the tensors are unchanged). Re-download, or reconvert. The teacher-forced checks above never saw it, because they feed token IDs directly.

Convert with this converter: it writes the grouped convolution's weights tap-major, which the graph expects, so GGUFs made by other converters do not load.

Verified on Strix Halo (Radeon 8060S):

Q4_K_M (5.17 GiB), by device:

F16 on Vulkan0: 1,138 tok/s prefill, 45.9 tok/s decode, perplexity 20.59.

* 512-token chunks over this repository's docs (PORTING.md, hrx.md, README.md), for comparing quants and devices, not models. ROCm's Q4 matmuls quantize activations to 8 bits, hence its slightly higher figure.

Vulkan decodes fastest, so auto and --device vulkan stay the default for ZAYA. The engine's --device rocm server is built from ROCmFPX's tree, which has no ZAYA; the ROCm numbers above are our llama.cpp built with GGML_HIP=ON for gfx1151.

HRX0 decoded at 25.5 tok/s until llama.cpp #24: ggml-hrx sent seven of ZAYA's ops per layer to the CPU (the grouped-conv matmul, the router's softmax, top-k and gather, and a few copies), 641 graph splits per decoded token. New HRX kernels for those ops (see HRX) make the decode graph one split, and decode 1.9x faster. The perplexity moves from 21.55 to 21.65, which is kernel rounding amplified by top-1 routing: with the new kernels turned off (GGML_HRX_DISABLE_DISPATCH) the build reproduces the old figure exactly. Vulkan and the CPU are unchanged.

What it took, besides the port: CCA's grouped convolution runs as one batched matmul per tap (as one small matmul per group it held Vulkan decode at 50 tok/s), and the graph avoids what HRX and ROCm lacked: copies into part of a state row, concatenating strided views, l2_norm, and a two-tap ssm_conv kernel on ROCm.

ZAYA1-74B-preview#

ZAYA1-74B-preview is the same design at 74.8B parameters and 60 layers. It alternates sliding-window layers (a 4,097-token window, rope theta 1e4) with full-attention layers (theta 1e7). The converter writes the window, the per-layer pattern and the second rope base (llama.cpp #11). ZAYA1-8B has no sliding layers and is unchanged: its Q4_K_M perplexity matches to four decimals, and its F16 still matches transformers FP32 at 95 of 96 teacher-forced positions.

python third_party/llama.cpp/convert_hf_to_gguf.py --remote Zyphra/ZAYA1-74B-preview --outtype q8_0 --outfile zaya1-74b-Q8_0.gguf
build/hrx/llama/bin/llama-quantize --allow-requantize zaya1-74b-Q8_0.gguf zaya1-74b-Q4_K_M.gguf Q4_K_M
1bit serve -m zaya1-74b-Q4_K_M.gguf --device vulkan --ctx-size 8192

--remote streams the safetensors from Hugging Face, so the 150 GB BF16 checkpoint never has to fit on disk. The Q8_0 file is 79.7 GB, and requantizing it to Q4_K_M needs --allow-requantize. GGUFs made with --remote before llama.cpp #14 have no chat template, because the download skipped chat_template.jinja. Reconvert them, or serve them with --chat-template-file.

Verified on Strix Halo, HF revision bc8eb466, Q4_K_M (42.6 GiB, 45.7 GB):

There is no reference check for the 74B: a 74B FP32 transformers run needs about 300 GB of memory, which does not fit on Strix Halo.

ZAYA1-base and ZAYA1-reasoning-base: legacy checkpoints#

ZAYA1-base, ZAYA1-reasoning-base and Zyphra's *-legacy repos keep the original Megatron-style checkpoint: - attention and MoE are separate half-layers; - sizes live in per-layer lists; - experts are stored one by one.

transformers cannot load them. The converter maps them to the transformers layout itself (llama.cpp #14). Converting Zyphra/ZAYA1-8B-legacy gives a GGUF whose 1,283 tensors are byte-identical to the one from Zyphra/ZAYA1-8B, and the model metadata is identical too.

Both base models are the 8B shape (40 layers, 16 experts), with rope theta 1e6 and a 32,768-token context. Checked on Strix Halo, Q4_K_M (5.56 GB), on Vulkan0:

* Raw-text perplexity favors base models; the post-trained ZAYA1-8B scores higher on it. These figures are not comparable with the docs-based perplexity in the table above. ZAYA1-reasoning-base answers chat prompts with its own template, thinking first.

Until llama.cpp #15, ZAYA returned wrong results whenever a ubatch held more than one sequence: llama-perplexity at its default batch, and 1bit serve answering concurrent requests. At four sequences per ubatch, ZAYA1-8B F16's wikitext perplexity was 58,259 against 27.89 with one; now both are 27.89 on the CPU.

ZAYA1-VL-8B: images#

ZAYA1-VL-8B is ZAYA1 with Qwen2.5-VL's vision tower and two additions (llama.cpp #18):

• Vision-only LoRA. Rank-8 LoRA on CCA's q, k, both value projections and o_proj, and rank 32 on every expert, used on image tokens only. mtmd decodes each image as its own ubatch of embeddings, so that ubatch takes the LoRA and text ubatches run the ZAYA1 graph unchanged.
• Bidirectional attention within an image. The mmproj key clip.vision.decode_non_causal makes mtmd decode the image non-causally. The whole image must fit one ubatch, so 1bit serve --mmproj passes -b 4096 -ub 4096 (Qwen2.5-VL caps an image at 4,096 tokens).

python third_party/llama.cpp/convert_hf_to_gguf.py <Zyphra/ZAYA1-VL-8B> --outtype f16 --outfile zaya1-vl-8b-F16.gguf
python third_party/llama.cpp/convert_hf_to_gguf.py <Zyphra/ZAYA1-VL-8B> --mmproj --outtype f16 --outfile mmproj-zaya1-vl-8b-F16.gguf
1bit serve -m zaya1-vl-8b-F16.gguf --mmproj mmproj-zaya1-vl-8b-F16.gguf --device vulkan --ctx-size 8192

Checked on Strix Halo against Zyphra's own code (their transformers branch zaya1-vl, FP32 on the CPU), teacher-forced on llama.cpp's greedy answers to three image questions (101 tokens). The prompts are identical: 208 tokens, with the image at the same position.

Zyphra's flash-attention path, which its paper describes, makes everything up to two tokens past the image bidirectional. llama.cpp decodes the image bidirectionally and the tokens around it causally.

On a synthetic image (a red square, a blue circle, "HELLO 42") the model reads the text exactly, and on a photo it recognises the New York Times front page of the moon landing. F16 decodes at 38-51 tok/s on Vulkan0. ZAYA1-8B is unchanged: 95/96 teacher-forced, the same wikitext perplexity.

OPT, GPT-Neo, CodeGen and GPT-J, from our llama.cpp#

Upstream llama.cpp has no model code for these four older families (for gptj it has only the name). Our llama.cpp ports them (PR #8, engine #138). 1bit serve --device vulkan routes them to that build's Vulkan0, the same way it routes ZAYA.

Each port was checked against transformers FP32: teacher-forced top-1 agreement over 3 prompts of 32 tokens, and one prompt of 340 tokens.

View this page's source on GitHub ↗