HRX: we put AMD's experimental IREE runtime inside our engine
AMD calls HRX "a lighter, more focused subset of ROCm" — an IREE-based HIP replacement shipped as a self-contained llama.cpp bundle. This week it moved inside our engine: dlopen'd in-process, token-level decode on the HRX device, two llama.cpps in one process without them colliding, and warm decode at ~2× HIP on the models its fused kernels can run. Plus the ceiling we measured, and the honest reframe that goes with it.
Milestone: AMD's experimental NPU runtime now runs inside our engine — one process, two runtimes, roughly twice the warm decode, with the honest ceiling measured. Full technical write-up → HRX backend research
What HRX actually is
HRX — Hip Runtime Extended — is AMD's experiment at a lighter, more focused subset of ROCm for client silicon (their words, in the upstream RFC). It's built on IREE (LLVM MLIR) with a Loom kernel substrate, and AMD ships it as a self-contained llama.cpp bundle: libllama.so + libggml-hrx.so + libhrx.so + libloomc.so, no ROCm install needed. We've run it as a subprocess backend for a week. This week it moved into our process.
Two llama.cpps in one process
The bundle ships a complete llama.h C API — 82 KB of it — and our engine already links its own llama.cpp for the Vulkan backend. Two copies of an unversioned C API in one binary is normally a recipe for silent memory corruption, so we did the boring careful thing: dlopen the bundle's libllama.so with RTLD_DEEPBIND, resolve every call through a dlsym'd function table, and static-assert the ABI structs (72/160/56 bytes) against the bundle's own headers. The two llama.cpps never see each other's symbols.
The result is a token-level HRX backend: generate() decodes one token at a time on the HRX device (gfx1151), all layers offloaded, KV cache in llama.cpp structures. No HTTP round trip, no subprocess — the binary we used to spawn is now just a library we call.
It works — and it's fast
End to end through 1bit unified on Qwen3-30B-A3B Q4_K_M (an 18.6 GB MoE GGUF): "What is the capital of France?" → "Paris", backend: hrx_gpu. Warm decode ~80–87 tok/s in-process, vs 38 tok/s for the same bundle spawned fresh as a subprocess, and ~70 tok/s for our HIP backend — the in-process path removes the server round trip and the fresh-start JIT cost, and HRX's fused kernels genuinely run warm decode at ~2× HIP for graphs they can fuse.
The ceiling, measured honestly
Then we found the wall, and measured it across six model families. HRX's fused node set does not include GET_ROWS — the token-embedding row-gather that starts every decode — for most quantization types. Q4_K embeddings fuse. q5_0, q8_0, Q4_K_S and IQ2XXS fail closed with llama_decode: ret = -3.
We tried everything local: n_gpu_layers tuning, pinning the embedding into a CPU buffer via buffer-type overrides. Both are dead ends in this fork — the scheduler keeps one graph split on HRX and no local knob forces the row-gather onto the CPU backend. The subprocess has the identical limitation, so being in-process costs us nothing. This is an upstream problem — llama.cpp PR #27218, still a draft — not ours to fix locally.
The honest reframe: HRX is an acceleration lane, not the engine. Our multi-lane router (HIP, Vulkan, FLM NPU, dedicated engines) with route-order failover means a model HRX can't fuse fails over to ggml_vulkan and completes anyway — verified end to end. The engine is complete with or without HRX.
Zero-copy is a separate, proven story
To keep the two halves of the vision straight: zero-DMA-copy on the NPU side is not part of the HRX lane. It's the SharedBO substrate, re-proven on this box the same week: a Vulkan compute shader reads the NPU's KV cache straight out of NPU-owned pages via dma-buf import and matches the CPU reference (rel err 2e-4). One page set, three views, no memcpy.
Lemonade, too
All 43 chat-capable lemonade models now carry -HRX registry variants. The ones with K-quant embeddings serve directly on HRX; the rest point at the Vulkan variant — and because lemond has no failover, we documented exactly which is which, so nobody gets a silent 500 on a model HRX can't fuse.
Run it
HRX_INPROCESS=1 build/1bit unified -m "Qwen3 30B A3B Instruct" — the bundle loads in-process, weights go to HRX0, warm decode runs at ~2× HIP. For models outside the fused set, the router does exactly what it was built to do: fail over and finish the answer.