Documentation
How to run the engine, what each device does, and how the pieces fit. Measured numbers are on the wiki.
Start
| Page | What it covers |
|---|
| 1bit serve | The engine runs inside Lemonade: Lemonade runs 1bit serve as a backend, the way it runs llama-server. 1bit serve exposes nothing but an OpenAI-compatible API, so it also works on… |
| Lemonade | The 1bit engine runs inside Lemonade. Lemonade is the server users talk to, with its own catalog, downloads, router and UI. For the models it serves with 1bit, it runs the engine… |
| Weekly releases | 1bit engine is pinned upstream: every backend it builds is a submodule at a fixed commit (thirdparty/), and the bump- workflows open a PR whenever an upstream moves. The engine… |
| How we measure | tools/bench.py measures 1bit serve configurations against each other. It is how the serve numbers in these docs are taken, and how a recipe earns its place. |
| Recipes | A setting that makes one model faster on one route can make another slower. For example, 1024-token micro-batches speed up a Hadamard-rotated MoE model's prompts by 3-8% and slow… |
Devices
| Page | What it covers |
|---|
| NPU | The fast lane runs one whole-layer kernel per decoder layer, plus an lm-head kernel. For each token, the host submits the 28 layer runs and the lm head as one XRT runlist. This… |
| HRX | On Strix Halo, HRX (AMD's ggml-hrx with our Loom kernels) reads Qwen3.8-27B prompts at 335 tok/s and decodes ZAYA1-8B at about 90 tok/s. It is the engine's GPU route: a… |
| Vulkan (upstream) | 1bit serve --device vulkan runs a llama-server built from upstream llama.cpp (MIT), pinned to its latest release in thirdparty/llama.cpp-vulkan, plus the few upstream pull… |
| Lean (ROCmFP4, ROCmI4) | 1bit serve --lean trades a little accuracy for speed: with a Hadamard-rotated file it reads Qwen3.8-27B prompts at 509 tok/s, against about 400 on the exact path. It runs models… |
| ZINC | ZINC (MIT) is a single-binary GGUF engine written in Zig. It has its own kernels for four GPU backends, and serves /health, /v1/models and /v1/chat/completions: |
| Apple Silicon | On macOS, 1bit serve --device mlx serves MLX models. The executor is the server binary of lemon-mlx-engine (fork bong-water-water-bong/lemon-mlx-engine). It builds MLX's Metal… |
Components
| Page | What it covers |
|---|
| MoE streaming | The goal: run mixture-of-experts models larger than Strix Halo's 128 GB. Only the experts a token routes to are needed. The hot ones stay in RAM, and the rest come from the NVMe… |
| Laya router | Laya picks where each request runs: it sorts each conversation into code, prose, short answer or long document (95.5% right), and a measured policy picks the device. The test set… |
| Model registry | Step 5 of the port: which Hugging Face models this engine can run, kept current every day. Two numbers are reported, and they are never added together: |
| Architecture gaps | The registry maps an HF architecture only when a pinned backend's own code accepts it. It is generated from the pinned sources; nothing in it is typed in by hand. So "support"… |
| Windows | 1bit.exe runs on Windows 10 and later (x64): 1bit serve with GGUF models on the CPU (--device cpu, which auto picks) and ONNX Runtime GenAI models (--device onnx, on the CPU),… |
| ONNX Runtime | 1bit serve --device onnx serves ONNX Runtime GenAI models: a directory with genaiconfig.json, model.onnx and the tokenizer, the format of Lemonade's ONNX models. The engine runs… |
| ComfyUI.cpp | ComfyUI.cpp runs ComfyUI workflows in C++. It is a port of ComfyUI's workflow engine and of the Stable Diffusion 1.5 nodes on LibTorch, pinned as the submodule… |
| Tokenizers | The engine reads any model's tokenizer.json through Hugging Face tokenizers, the Rust library behind the tokenizers npm and PyPI packages, at the same release as those packages. |
| Linux kernel | The engine's NPU driver (amdxdna) and GPU driver (amdgpu) come from the kernel, so the kernel is pinned like every other dependency: thirdparty/linux is upstream torvalds/linux at… |
Project
| Page | What it covers |
|---|
| Porting map | This repository is the working 1bit-MONSTER engine, ported without its history. Each step below is one PR (or a short series) that builds and runs on Strix Halo before the next… |
More
| Page | What it covers |
|---|
| DwarfStar | DwarfStar (antirez/ds4, MIT) is a native inference engine written for a few large MoE models: DeepSeek V4 Flash (and V4.1 Flash and PRO on bigger machines), GLM 5.2/5.3 Flash and… |