ROCm vs CUDA, from the 1-bit trenches
We ship on ROCm, not CUDA, on purpose. Real kernels, real benchmarks, and the 1-bit reason AMD's stack wins for our use case.
Milestone: 1-bit matmuls on AMD silicon measured against CUDA from real kernel work — the comparison behind why every model runs on AMD's stack. Full technical write-up → engine comparison report
The setup
The engine's backends run on the Ryzen AI NPU (Strix Halo, Kraken Point) and ROCm GPUs — with Vulkan and CPU as the fallback layer. We've shipped to real silicon on both AMD stacks, so the comparison below is from our kernel logs, not a vendor deck.
Where ROCm wins for us
1-bit arithmetic. Our packed Q4NX path is addition-dominated — the kind of kernel NPU silicon is literally built for. The measured sprint on the NPU (244 → 3.4 ms/tok) came from matching dispatch to what the silicon does natively, and ROCm gives us the same direct-to-hardware access on dGPUs without a translation layer.
No cloud tax. CUDA is the default because the datacenter is NVIDIA's. Local inference has no such gravity — the machine on your desk is more likely AMD, and the NPU in it is AMD silicon with first-class support in the engine.
One binary, both paths. The engine detects the backend at load: hip_1bp_gpu for the 1-bit GPU path, backend_npu for the NPU, cpu_generic when neither is present. Same weights, same API.
Where CUDA still leads (and we don't care)
For training at scale, CUDA's ecosystem is unmatched — but the engine doesn't train. For inference on hardware you own, the gap is a story about datacenters, not about your laptop. 100% of HuggingFace models load either way; we'd rather run on the chip you have.
The numbers that matter
Qwen3.6-35B-A3B streams fully on the Ryzen AI NPU: 13.65 tok/s decode and 221.96 tok/s prefill per AMD's official FastFlowLM figures, with our independent first pass at 75.65 tok/s via the Vulkan path on Strix Halo. That's a 35B-class model on a laptop's built-in NPU — the ROCm world's answer to 'CUDA only'.