1bit serve: the engine behind one OpenAI-compatible API#

The engine runs inside Lemonade (Lemonade): Lemonade runs 1bit serve as a backend, the way it runs llama-server. 1bit serve exposes nothing but an OpenAI-compatible API, so it also works on its own with any OpenAI client.

1bit serve -m <model> [--port 8000] [--host 127.0.0.1]
           [--device auto|npu|hrx|rocm|cpu|vulkan|zinc|ds4|mlx] [--ctx-size N] [--alias NAME]
           [--llama-server PATH] [--zinc PATH] [--ds4 PATH] [--ssd-streaming] [--hrx-libhsa PATH] [--mlx-server PATH]
           [--prefill-device hrx] [--prefill-min-tokens N] [--lean]
           [--mtp HEAD.gguf | --dflash DRAFT.gguf] [--mtp-max N] [--mtp-p-min P] [--mmproj MMPROJ.gguf]
           [--laya | --laya-model DIR] [--route-policy FILE]
           [--moe-slots N|auto] [--moe-subst R] [--moe-prefetch N]
           [--parallel N] [--adaptive] [--adaptive-at N]
           [--embed MODEL.gguf] [--rerank MODEL.gguf]
           [--npu-opt KEY=VALUE ...]

One model per process:

"llama-server devices" means vulkan, hrx and rocm. These routes go to the llama-server behind the model, which is how Lemonade's llamacpp backend reaches them (the onebit recipe inherits it). On npu, zinc, ds4 and mlx they answer 501.

Where the model runs#

A build without the private add-on answers a Qwen3.6-35B-A3B directory with "the Qwen3.6-35B-A3B NPU route is not part of this build; build with -DONEBIT_NPU_PRIVATE=". --npu-opt KEY=VALUE passes an option through to a private route. On a route that keeps its cache between requests, usage.prompt_tokens_details.cached_tokens counts the reused prompt tokens and timings.cache says where they came from. On the NPU a streamed request with stream_options.include_usage gets a last chunk with usage and timings, as OpenAI's API sends it.

chat_template_kwargs.enable_thinking: false works on every device. The GPU backends apply the model's own chat template. The NPU route emits what Qwen3's template does: an empty think block after the assistant prefix. For Qwen3.6-35B-A3B it also opens <think>\n when thinking is on, as that model's template does.

For a .gguf the engine starts that server as a private child on a loopback port and forwards the OpenAI routes to it, streaming included. Replies carry the served model name. For ZINC, which rejects foreign model ids, requests go out without model. auto means HRX for GGUF: Vulkan is leaving the engine (RFC #213), and HRX decodes within 10% of it on the gate models (docs/hrx.md). With --laya (or --laya-model DIR), Laya classifies each conversation and the route policy picks the device: the built-in policy sends every class to HRX, long documents included, since HRX prefills Qwen3.8-27B at 335 tok/s pp512 (docs/laya.md). The first turn of a conversation pays about 0.5 s for the decision.

HRX needs TheRock's HSA runtime: the distro libhsa rejects gfx1151's PM4-emulation probe, and then HRX registers no device. serve sets IREE_HAL_AMDGPU_LIBHSA_PATH itself unless you did. It uses --hrx-libhsa, else the build's copy, else the first one under /opt/rocm-therock.

The child binaries default to this build's (-DONEBIT_HRX, -DONEBIT_ZINC, -DONEBIT_DS4), then $ONEBIT_LLAMA_SERVER / $ONEBIT_ZINC / $ONEBIT_DS4, then llama-server / zinc / ds4-server on PATH.

Images (--mmproj)#

--mmproj <mmproj.gguf> hands llama-server a vision projector on the llama.cpp routes (vulkan, hrx, rocm). Chat messages may then carry image_url parts (a data: URL or a file URL), which llama.cpp's mtmd encodes and places in the prompt. The mmproj comes from convert_hf_to_gguf.py --mmproj on the same checkpoint as the model. With it, llama-server runs with -b 4096 -ub 4096: an image is decoded as one ubatch, which models that attend to an image bidirectionally (ZAYA1-VL, Gemma 3) need. Zyphra's Zamba2-VL and ZAYA1-VL-8B are in Vulkan (upstream). The NPU, ZINC, DwarfStar, MLX and ONNX routes take no --mmproj.

Multi-token prediction (--mtp)#

--mtp <head.gguf> turns on llama-server's draft-mtp speculative decoding: the model's own MTP head (Unsloth ships it as MTP/mtp-<model>-*.gguf) drafts tokens on the same device, and the model checks them in one batch. --mtp-max N caps the draft length; --mtp-p-min P drafts a token only when the head is at least that sure of it. It works on the llama.cpp devices (vulkan, hrx, rocm).

Measured through 1bit serve on Strix Halo, Qwen3.8-27B UD-Q4_K_XL with its Q4_0 MTP head, decode tok/s on three chat prompts (code / prose / short), 2026-09-24:

Vulkan with MTP is the pick: 2.4-2.9x on every prompt, same file, same accuracy (a drafted token is kept only when the model agrees). ROCm edges it on code only.

The full sweep on Vulkan (draft length x --mtp-p-min, code / prose / short; 12.2 / 12.3 / 12.3 without MTP):

Use the default for chat and --mtp-max 6 --mtp-p-min 0.5 for code (3.4x): code is predictable enough to keep long drafts, prose is not. Draft length 8 collapses on every prompt. Adding n-gram drafting to MTP gains nothing. Small-active MoE models are the opposite case: on Qwen3-Coder-30B-A3B (3B active, 88 tok/s on Vulkan) every draft model tried (Qwen3 0.6B / 1.7B / 4B) was slower than no drafting, even at 82-87% acceptance.

Qwen3.8-Flash-Next: drafting from part of the vocabulary#

Flash-Next's MTP head computes logits over all 248,320 tokens for every drafted token. That is the model's 644 MiB Q8_0 output matrix, read three times per decode step at draft length 3. A draft only proposes; the model checks every token, so the head can be cut down to the tokens it is likely to propose. tools/mtp_draft_vocab.py writes such a head from Unsloth's shared head file and the model's own output rows. Our llama.cpp fork (qwen4exp, src/models/qwen4exp-draft-vocab.cpp) runs it:

python3 tools/mtp_draft_vocab.py \
  --mtp MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --target UD-Q4_K_XL/ \
  --ids config/draft-vocab/qwen38-en-code-65536.txt --out mtp-Qwen3.8-Flash-Next-dv65k.gguf

config/draft-vocab/qwen38-en-code-65536.txt is dime-online's 65,536-id list, picked from English prose and source code (qwen3.8-Flash-DGX-UltraFast, Apache-2.0, see NOTICE). It makes the head 170 MiB.

Measured with the fork's Vulkan llama-server on Strix Halo, 2026-10-01. Settings: - UD-Q4_K_XL, greedy, draft length 3, -lm mmap -lzm on - decode tok/s, median of rounds 2-3 of 3

• English and code: 8-9% faster, the same acceptance, and about 6 ms less per decode step (76 -> 70 ms).
• Other languages: text outside the list loses, as the French row shows. Use the full head for such text.
• Output text: decoding with a draft is not token-identical to decoding without one, even with the full head. Batched verification flips near-ties (one flip each in prose and long code here). The cut head gives the full head's text in rounds 2-3, with two other flips in round 1.
• Q4_K_M: Unsloth's shared Q4_K_M head was no faster and flipped more often.
• -lm mmap: Flash-Next needs it on this llama.cpp. Without it, the 27.5 GiB per-layer embedding table is copied into RAM next to the ~82 GB the GPU holds, which does not fit in 128 GB.

DFlash draft models (--dflash)#

--dflash <draft.gguf> uses a DFlash block-diffusion draft model instead of the MTP head: the draft emits a whole block of tokens in one pass, reading the model's hidden states, and the model checks the block in one batch. It turns on llama-server's draft-dflash (DFlash2's candidate selector included). A drafter is trained for one target model; z-lab publishes them on Hugging Face (for example z-lab/Qwen3.8-27B-DFlash2, 2B, Apache-2.0). --dflash and --mtp are exclusive. --mtp-max and --mtp-p-min set the draft length and threshold for either; without --mtp-max, --dflash drafts full blocks (llama-server's own default of 3 throws most of the gain away).

Convert the drafter with upstream llama.cpp's converter. It needs only the target's config.json and tokenizer files, not its weights:

hf download Qwen/Qwen3.8-27B --include "*.json" "*.jinja" "merges.txt" "vocab.json" --local-dir Qwen3.8-27B-hf
hf download z-lab/Qwen3.8-27B-DFlash2 --local-dir DFlash2
python convert_hf_to_gguf.py DFlash2 --target-model-dir Qwen3.8-27B-hf --outtype bf16 --outfile dflash2-bf16.gguf
llama-quantize dflash2-bf16.gguf dflash2-q8_0.gguf Q8_0
1bit serve -m Qwen3.8-27B-Q4_0.gguf --device vulkan --dflash dflash2-q8_0.gguf

Measured through 1bit serve --device vulkan on Strix Halo, Qwen3.8-27B Q4_0 (Unsloth), the same three chat prompts as above (code / prose / short, decode tok/s), runs back to back on a shared box, 2026-09-28:

DFlash2 is the pick for code (4.1x, +44% over MTP) and ties or beats MTP on prose. MTP wins short answers (the translation prompt stops after ~16 tokens, too few to fill blocks). The Q8_0 drafter is faster than BF16 (42.5 vs 38.9 on code, direct llama-server). A drafted token is kept only when the model agrees, so the output is the model's own.

--dflash also works on the lean ROCm route: a Hadamard-rotated Q4_0 file gets W4A4 prompt processing and DFlash2 decode from one server, 518 t/s prompt and 46.0 / 28.4 / 17.6 tok/s decode (Lean (ROCmFP4, ROCmI4)). --dflash gets --spec-draft-p-min from a recipe (0.4 on rocm, 0 elsewhere) unless --mtp-p-min is given, and without --mtp-max drafts the drafter's block minus one (its dflash.block_size; 16 when the file does not say). Any drafter type works next to a Hadamard-rotated file, plain Q4_0 included: the backend rotates the activations of the rotated file's weights only. Qwen3.8-27B-H32 with a Q4_0 DFlash2 drafter runs as fast as with the Q8_0 one.

Recipes (--recipes, --no-recipes)#

Tuned llama-server settings that depend on the model and route, such as a MoE file's micro-batch size or a DFlash drafter's p-min, are recipes in config/recipes.json. Each one carries the measurement behind it. Serve prints each recipe it applies. Anything serve sets itself wins over a recipe. --recipes FILE replaces the built-in set and --no-recipes turns recipes off (Recipes). To measure a setting before making it a recipe, use tools/bench.py.

MoE models larger than memory (--moe-slots)#

--moe-slots N (with --device vulkan) keeps a MoE model's routed experts in the file and holds N of them, across all layers, in GPU memory. Decode reads the missing ones from the drive as the router picks them. The rest of the model loads on Vulkan0 as usual.

1bit serve -m Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf --moe-slots 4608 --ctx-size 8192

Qwen3-Coder-30B has 6,144 experts; at 4,608 slots it decodes 39 tok/s warm, at 1,536 about 6-7 (docs/moe-streaming.md, "Streaming in the inference path"). A model that fits in memory is faster without it (79-91 tok/s resident). It works with the Vulkan pin's llama-server only, not with --prefill-device.

--moe-subst R trades a little accuracy for fewer reads: when the router picks an expert that is not in memory, a resident one among its next choices takes its place if it scores at least R times as high. 0.5 cut Coder-30B's drive reads by 20-27% and sped decode up 10-35% at 1,536 and 3,072 slots, for a KL divergence of 0.008-0.012 against the exact model (docs/moe-streaming.md, "Routing that prefers resident experts"). Without it, routing is exact.

Qwen3.8-Flash-Next UD-Q4_K_XL (111 GB) runs this way from 18-32 GiB of GPU memory: 3.5-6 tok/s exact at 4,608-9,216 slots. On that model use --moe-subst 0.9 or none: 0.5 costs a KL divergence of 0.07 there.

--moe-slots auto sizes the cache from free memory: MemAvailable, less the other tensors the GPU holds and a reserve (8 GiB or 15%), divided by the model's mean expert size. On a box with 40 GiB free it gave Flash-Next 9,331 of 24,576 experts (27.2 GiB) beside 5.1 GiB of other tensors, and serve held 145 MiB of anonymous memory. Other processes can take that memory back later; on a shared box, give a number instead. The gate-ahead prefetch is off unless --moe-prefetch N asks for it (it doubled Flash-Next's decode time).

Prompts are slow when streaming: a batch larger than 8 tokens runs the experts on the CPU from the mapped file (Flash-Next: 0.4-0.8 tok/s of prompt). Keep prompts short, or see docs/moe-streaming.md, Next.

Many requests at once (--parallel)#

--parallel N gives llama-server N slots (-np N): requests that arrive together decode together, one read of the weights per step for all of them (continuous batching, what vLLM is built on). --ctx-size is split across the slots, so size it for all of them.

Total decode tok/s with 1-16 simultaneous 256-token requests, 16 slots, 2026-09-24:

Vulkan peaks at 8 requests and falls back at 16; ROCm keeps scaling to 16, where it serves 5.8x (27B) and 3.7x (Coder) the best single stream. Use --device vulkan for up to about 8 concurrent users, --device rocm --parallel 16 beyond that. Two backends decoding at once, by comparison, add 18% (docs/lean.md).

Growing with the load (--adaptive)#

No single setting wins at every load. One request is fastest on Vulkan with --mtp; up to 8 are fastest batched on Vulkan; past that ROCm keeps scaling (the tables above). And MTP and batching do not mix: a server with MTP loaded batches at about two thirds of the throughput (Qwen3.8-27B, 4 requests: 24.9 tok/s with MTP loaded, 36.9 without), even with each request's draft length set to 0.

--adaptive runs two backends, the model loaded on each: Vulkan with --adaptive-at slots (and --mtp if given) and ROCm with 16 slots (never MTP). A request goes to Vulkan while Vulkan holds fewer than --adaptive-at requests, and to ROCm otherwise; choosing and reserving the slot is one locked step.

Qwen3.8-27B UD-Q4_K_XL, total tok/s by simultaneous requests (2026-09-24):

• Serving many users: --adaptive --adaptive-at 8. It matches Vulkan batching up to 8 requests and keeps climbing past it (63.5 at 24, where Vulkan alone falls back). At 12-16 the two backends contend for the GPU and it dips below ROCm alone.
• One user at a time: --mtp without --adaptive (35-42 tok/s). --adaptive --mtp gives the lone request MTP speed but trails at every larger load.

1bit serve -m Qwen3.8-27B-UD-Q4_K_XL.gguf --adaptive --adaptive-at 8 --ctx-size 65536

It needs both builds (ONEBIT_VULKAN, and ONEBIT_LEAN + ONEBIT_LEAN_ROCM) and memory for two copies of the model; --ctx-size applies to each backend and is split across its slots. On exit, serve logs how many requests each backend took.

Short and long prompts (--long-model)#

What a user waits for is the first token plus the answer. With a short prompt the faster decode wins (Vulkan); with a long one the faster prefill does (the W4A4 route, from about 2,000 prompt tokens; Lean (ROCmFP4, ROCmI4)). --long-model serves the same model both ways: -m on its usual route, and a Hadamard-rotated Q4_0 of it (tools/hadamard_q4_0.py) on ROCm with W4A4. A conversation whose first request has --long-from prompt tokens or more (default 2048; chats are counted through the model's template) goes to the rotated file, the rest to -m. A conversation stays where its first request went, so its prompt cache stays useful as it grows. Each response carries X-1bit-Route: short|long <tokens> <device>.

1bit serve -m Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf --long-model Qwen3-Coder-30B-A3B-Q4_0-H32.gguf --ctx-size 20480

The rotated file is published as 1bit-MONSTER/Qwen3-Coder-30B-A3B-Q4_0-H32-GGUF; tools/hadamard_q4_0.py makes one for another model.

Qwen3-Coder-30B-A3B, 256 output tokens, effective tok/s (output tokens over the whole wait; tools/e2e_bench.py, 2026-09-28):

The --long-model row ran while another job used the box's memory bus, which cost its long side 2-5% against the W4A4 row. A rotated MoE file now runs 1024-token micro-batches (-ub 1024; a dense one keeps 512, Lean (ROCmFP4, ROCmI4)), which raises the W4A4 prefill 6-10% (server-reported 2,217 / 1,742 / 1,295 tok/s at 2K / 8K / 16K, from 2,000 / 1,607 / 1,221): a 16K-token prompt's first token at 12.96 s instead of 13.89. Both files stay loaded (here 17 + 16 GiB). It needs the Vulkan build and ONEBIT_LEAN + ONEBIT_LEAN_ROCM, and -m on a llama-server route (vulkan, hrx or rocm); it does not combine with --laya, --adaptive or --mmproj.

RAG (--embed, --rerank)#

--embed MODEL.gguf serves /v1/embeddings and --rerank MODEL.gguf serves /v1/rerank, each from its own llama-server beside the chat model (--embedding, --reranking, the whole input in one batch). In a build with HRX that server runs on HRX0 with one slot and inputs of up to 2048 tokens (HRX runs one sequence per batch); a build without HRX keeps Vulkan, 4 slots and 8192 tokens. /v1/models lists them with their role. A RAG client embeds its documents, retrieves by cosine similarity, reranks the best few and asks the chat model with them as context, all against the one server:

--embedding and --reranking serve the model itself in that role, as Lemonade loads embedding and reranking models: 1bit serve -m nomic-embed-text-v2-moe.Q8_0.gguf --embedding. They run on vulkan, hrx or rocm. On Strix Halo (2026-10-02, HRX0 against Vulkan0, two repeats each): Qwen3-Embedding-0.6B Q8_0 embeddings repeat bit for bit on HRX, cosine 0.99961-0.99988 to Vulkan's; bge-reranker-v2-m3 Q8_0 scores repeat exactly and rank the same as Vulkan's (8.609 / -6.756 / -0.401 / -11.020 against 8.614 / -6.757 / -0.361 / -11.019). jina-reranker-v1-tiny still fails on HRX: its JIT can't link the fp32 matmul-with-bias kernel it needs.

1bit serve -m Qwen3.8-27B-UD-Q4_K_XL.gguf --mtp mtp-Qwen3.8-27B-Q4_0.gguf \
  --embed Qwen3-Embedding-0.6B-Q8_0.gguf --rerank qwen3-reranker-0.6b-q8_0.gguf

Measured end to end on Strix Halo (2026-09-24): the engine's own docs and README as the corpus (93 passages of about 120 words), four questions with known answers, Qwen3.8-27B UD-Q4_K_XL with --mtp answering, Qwen3-Embedding-0.6B Q8_0 embedding (queries prefixed with its Instruct: ... Query: line):

So use embeddings with a wide context and no reranker: retrieval is about 25 ms, and a few thousand extra prompt tokens cost little at 270-320 tok/s prefill. --rerank stays for larger corpora, where cutting 50 candidates to 8 matters more.

Long retrieved contexts are where --prefill-device hrx pays (26% on an 8192-token request, above).

Verified (Strix Halo, 2026-09-23)#

tests/serve_e2e.sh with Qwen3-0.6B: the Q4_K_M GGUF for the GPU devices and the Q4NX model directory for the NPU. The test checks /health 200, /v1/models, a chat that answers "Paris." under the served name, and streaming:

Qwen3.6-35B-A3B on the NPU is tested by the private add-on (docs/npu.md, "Private routes"). Without it, serve on that directory exits with the message above.

ctest runs these as serve_e2e_<device> when configured with -DONEBIT_SERVE_TEST_GGUF=<gguf> (and serve_e2e_mlx on macOS with -DONEBIT_MLX_SERVER). smoke_serve runs everywhere, CI included: tests/fake_backend.py stands in for llama-server. It checks the proxy (/health, /v1/models, the reply rename, SSE relay) and that no backend outlives a SIGKILLed serve.

View this page's source on GitHub ↗