What 1-bit actually does to a model
1-bit isn't a gimmick and it isn't free. Here's the real trade: what quantization to 1.58-bit does to weights, memory, and quality — measured, not projected.
Milestone: models packed to ~1.6 bits per weight run on hardware you already own — memory drops ~16x and matmuls become additions. What 1-bit actually does to a model, explained plainly. Full technical write-up → journey.md — the 1-bit numbers
The idea: weights that fit in 2 bits
A standard fp16 weight is 16 bits. A 4-bit quantized weight (GGUF Q4) is 4 bits. A 1-bit model pushes further: weights become ternary — {-1, 0, +1} — which needs just 1.58 bits on average. Memory drops roughly 16× vs fp16, 2.5× vs Q4.
The bigger win is arithmetic. Multiplying by a ternary weight is an addition or subtraction — no multiply needed at all. The matmul becomes pure accumulation, which is exactly what NPU hardware is built for. That's the 1-bit engine's core bet.
The cost, honestly
Ternary weights lose information. A model that's trained to be ternary (like BitNet-style training) absorbs the constraint during training and keeps quality high. One that's simply converted after the fact loses more. That's why the engine supports a spectrum — GGUF native, Q4NX 1BP, INT8 — and lets the model tell you which it wants.
The measured numbers
| Path | Format | Measured |
|---|---|---|
| NPU decode | Q4NX 1BP | 3.4 ms/tok |
| NPU decode (start) | Q4NX 1BP | 244 ms/tok |
| CPU prefill | GGUF | bit-exact vs llama.cpp |
The 70× NPU improvement across the sprint (244 → 3.4 ms/tok) is the same silicon, the same weights — the gain is dispatch, fusion, and the 1-bit arithmetic path, not a hardware upgrade.
Where it lands
1-bit is the laptop-class story: models that should need a datacenter running on the NPU you already own. It's also the gguf_to_q4nx one-liner: take any GGUF, pack it to 1-bit, and load it in the same binary. The conversion is part of the engine, not a separate tool.