research · field notes
Public notes on the work. Papers we steal from, benchmarks we run, things we ship. The earliest posts carry measured figures we no longer trust, and each one is marked with a correction.
serial · 2026
The embedded Lemonade server core is re-vendored to v11.9.0 — allowed_origins/CORS hardening, llama.cpp to b10723, the streaming-stall fix. Upstream now carries the HRX backend we built, so the re-vendor collapsed to a clean merge: we dropped our stream_stall_timeout and kept the newer hrx-b66 pin.
Zyphra Zaya 8B, 4-bit Q4NX, running entirely on the Strix Halo NPU via the HRX lane — decode 8.4 t/s (7× faster than the original dispatch), served over HTTP. Rounds 26-28: the F32-twin proof that the corr gap is summation order, the real Qwen3-0.6B on the XDNA 2 NPU (29.6/92.8 t/s), and then byte-identity — Qwen3-0.6B prefill logits corr 1.00000 vs the real FastFlowLM runtime on the same xclbin. Plus the memory model: one heap, one chained API call per layer, SharedBO pages with zero copies — the Windows NPU (MCDM) model matched on Linux.
The embedded Lemonade server core is re-vendored to v11.8.1 — upstream sync landed, and the re-vendor loop keeps every release in step. Engine HF coverage: 569 architecture tokens, 2,030 HF arch strings, 326,992/327,349 checkpoints mapped.
AMD's IREE-based HIP replacement is now an in-process, token-level decode lane — dlopen'd bundle, RTLD_DEEPBIND, ~2x HIP warm decode on fused models, the GET_ROWS ceiling measured across six families, and the honest reframe: an acceleration lane, not the engine.
AMD's 15-backend Lemonade release is now compiled into our engine — and the new one is built for our hardware: DwarfStar (ds4) runs DeepSeek V4 Flash on Strix Halo's gfx1151, ROCm runtime bundled, no system install. Plus download resume + rate limits, cloud wire_format passthrough, and our own merged PR in the changelog.
Every architecture token on HuggingFace — 569 of them — resolves to one binary. 326,992 checkpoints map to a single 1-bit engine. The census, the token catalog, and the formats: GGUF native, Q4NX 1-bit, INT8.
The engine now serves /v1/embeddings. Full local RAG — embed, index, retrieve, generate — on one binary, no cloud, no Python. This site's semantic search runs on it.
1-bit isn't a gimmick and it isn't free. The real trade of ternary quantization: weights become {-1, 0, +1}, memory drops ~16×, matmuls become additions — measured, not projected.
We ship on ROCm, not CUDA, on purpose. Kernel-level comparison from the engine's logs: 1-bit matmuls, NPU offload, and why AMD's stack wins for local inference.
The embedded Lemonade server core is re-vendored to v11.7.0 — new models, register/options/stats/metrics endpoints live, and a repeatable re-vendor loop so every upstream release lands without touching engine code. The engine's own HF coverage stays at 99.89%: 569 architecture tokens, 2,030 HF arch strings, 326,992/327,349 checkpoints mapped.
A two-node agent fleet on home hardware: two DeepSeek Harness agents on one LAN, both fed by 1bit.MONSTER. 35B at 166 tok/s on a Radeon 8060S, 8B at 96 tok/s on an RX 9070 XT, plus Qwen3-VL vision. Six engine bugs fixed along the way, with methodology.
AMD's official FastFlowLM numbers, decoded: 13.65 tok/s decode and 221.96 tok/s prefill fully on the NPU (Kraken Point), verified against the official Qwen config. Our own first pass: 75.65 tok/s via llama.cpp Vulkan on Strix Halo.
We extracted AMD's entire NPU model zoo: all 37 pre-built FLM models from ROCm/FastFlowLM v0.9.46, including the Qwen3.5 Omni multi-modal C++ source and Qwen3.6-MoE-35B with 256 experts. Auto-detection straight from the Q4NX header.
Zamba2 hybrid SSM and ZR1 reasoning, all 1BP. All four models converted and validated end-to-end on ZINC GPU: Zamba2 1.2B/2.7B/7B (Mamba2-hybrid) and ZR1-1.5B, 26 tok/s on Strix Halo. Two architectures, one engine.
Ternary and binary inference on NPU + GPU. Full support for Q1_0, TQ1, TQ2, IQ1_S/M and BitNet GGUF on both GPU (HIP/Vulkan) and NPU (XDNA 2), verified exact on real Strix Halo hardware. 4,200 lines across 31 files.
Train and fine-tune LLMs on Radeon, Instinct and Ryzen AI. Native AMD GPU support for training, fine-tuning and RL: 2x faster, 70% less VRAM, runs in 3GB. Complementary to our inference side.
Open-source token-level routing for multi-backend LLM inference on AMD Strix Halo. NPU draft plus GPU verify, with real logprobs on the wire.
AMD Strix Halo ships a 50 TOPS NPU. AMD's FLM runtime gets 11 tok/s and ROCm gets 41 tok/s; we drive the same NPU through a fused-layer engine. Throughput here is historical and unsourced.
DSpark was projected to hit 572 tok/s on Strix Halo. End-to-end measurement on the NPU disproved it: 0.1 to 0.2 tok/s at 0% draft acceptance. How and why, and what it would take to fix. Figures here are historical and unsourced.
The 97 tok/s number was real. The output behind it never was. Three silent bugs, LM head substitution, weight transpose and activation clipping, were producing garbage at full speed.
How a single tiny C++ binary auto-detects 73+ models across six backends, NPU, ROCm, Vulkan, CUDA, Metal and CPU, from one Q4NX header parse.
Seven engine versions in four days took NPU inference from 1,930 ms/tok to 3.4 ms/tok, a 72x speedup through batch decode, fused dispatch and INT8 GEMM. Throughput here is historical and unsourced.
Benchmarking the XDNA 2 NPU: historical NPU benchmark notes from the first month of the project. Figures quarantined as unsourced.
How QKV, attention and FFN combine into a single NPU dispatch, eliminating five of every six per-layer xclbin calls. Throughput here is historical and unsourced.
One person, a free Chess license and a C++ compiler turned AMD's locked-down NPU into an open-source inference engine. Throughput figures here are historical and unsourced.
A pi.dev-compatible coding agent CLI with 7 commands, NPU-native inference, package management, extensions, skills, themes and a systemd service. Ships as part of the 1bit MONSTER monorepo.
A 24x speedup in one session, from 244 to 10 ms/tok, beating FLM Kraken Point by 46%. OpenMP attention plus LM head. Full C++23, zero Python.
An early local run with one Lemonade endpoint over two compute lanes, iGPU llama.cpp ROCm plus NPU FLM, on Linux. Historical local run, kept for the record.
The notes will land. The engine is already there and it is free.
Read the source on GitHub