1bit engine
One engine behind an OpenAI-compatible API, running inside Lemonade: HRX on the Radeon iGPU with our Loom kernels, the XDNA 2 NPU, ZINC for NVIDIA and Apple GPUs, MLX on Apple Silicon.
latest · 2026-09-29Where the engine is going: HRX and the NPU. The engine is heading to AMD's HRX backend, whose kernels we write in Loom, plus the NPU.read the post →What the engine is, what runs where, and who it is built on.
engineDocs →Every device, every component, and how to run them.
docsBlog →The build log: what moved over, what got faster, what broke.
blogBenchmarks →Measured, not projected. Every number with how it was taken.
wiki1bit serve →The engine runs inside Lemonade: Lemonade runs 1bit serve as a backend, the way it runs llama-server. 1bit serve…
serveLemonade →The 1bit engine runs inside Lemonade. Lemonade is the server users talk to, with its own catalog, downloads, router and…
lemonadeWeekly releases →1bit engine is pinned upstream: every backend it builds is a submodule at a fixed commit (thirdparty/), and the bump-…
releasesHow we measure →tools/bench.py measures 1bit serve configurations against each other. It is how the serve numbers in these docs are…
benchRecipes →A setting that makes one model faster on one route can make another slower. For example, 1024-token micro-batches speed…
recipesNPU →The fast lane runs one whole-layer kernel per decoder layer, plus an lm-head kernel. For each token, the host submits…
npuHRX →On Strix Halo, HRX (AMD's ggml-hrx with our Loom kernels) reads Qwen3.8-27B prompts at 335 tok/s and decodes ZAYA1-8B…
hrxVulkan (upstream) →1bit serve --device vulkan runs a llama-server built from upstream llama.cpp (MIT), pinned to its latest release in…
vulkanLean (ROCmFP4, ROCmI4) →1bit serve --lean trades a little accuracy for speed: with a Hadamard-rotated file it reads Qwen3.8-27B prompts at 509…
leanZINC →ZINC (MIT) is a single-binary GGUF engine written in Zig. It has its own kernels for four GPU backends, and serves…
zincApple Silicon →On macOS, 1bit serve --device mlx serves MLX models. The executor is the server binary of lemon-mlx-engine (fork…
appleMoE streaming →The goal: run mixture-of-experts models larger than Strix Halo's 128 GB. Only the experts a token routes to are needed.…
moe-streamingLaya router →Laya picks where each request runs: it sorts each conversation into code, prose, short answer or long document (95.5%…
layaModel registry →Step 5 of the port: which Hugging Face models this engine can run, kept current every day. Two numbers are reported,…
registryArchitecture gaps →The registry maps an HF architecture only when a pinned backend's own code accepts it. It is generated from the pinned…
arch-gapsWindows →1bit.exe runs on Windows 10 and later (x64): 1bit serve with GGUF models on the CPU (--device cpu, which auto picks)…
windowsONNX Runtime →1bit serve --device onnx serves ONNX Runtime GenAI models: a directory with genaiconfig.json, model.onnx and the…
onnxComfyUI.cpp →ComfyUI.cpp runs ComfyUI workflows in C++. It is a port of ComfyUI's workflow engine and of the Stable Diffusion 1.5…
comfyuiTokenizers →The engine reads any model's tokenizer.json through Hugging Face tokenizers, the Rust library behind the tokenizers npm…
tokenizersLinux kernel →The engine's NPU driver (amdxdna) and GPU driver (amdgpu) come from the kernel, so the kernel is pinned like every…
kernelPorting map →This repository is the working 1bit-MONSTER engine, ported without its history. Each step below is one PR (or a short…
PORTINGDwarfStar →DwarfStar (antirez/ds4, MIT) is a native inference engine written for a few large MoE models: DeepSeek V4 Flash (and…
dwarfstar