MoE streaming from NVMe#

The goal: run mixture-of-experts models larger than Strix Halo's 128 GB. Only the experts a token routes to are needed. The hot ones stay in RAM, and the rest come from the NVMe drive as the router asks for them. The work splits three ways: - the dense layers (attention, shared experts, embeddings) on the GPU (HRX or Vulkan); - the routed experts wherever they run fastest: NPU (native int8 x int4) or GPU; - the NVMe feeding the cache.

This page holds the measurements, the expert cache (moe/, 1bit moe-cache), streaming in decode (1bit serve --moe-slots) and the tools that produced them (tools/moe_trace.cpp, moe_policy.cpp, moe_expert_bench.cpp).

What one token needs#

Qwen3.8-Flash-Next (qwen4exp) UD-Q4_K_XL, from its GGUF metadata:

A token reads 48 × 10 × 3.06 MiB = 1.47 GB of routed experts. The shared weights stay resident.

The drive#

Strix Halo's NVMe (YMTC PC41Q 2 TB), O_DIRECT reads from a model file:

With nothing cached, 1.47 GB per token at about 3.7 GB/s caps decode at about 2.5 tok/s. So the cache decides the speed.

On 2026-09-26, with other tenants active (load average 6-9), the same drive gave 1.2-4.2 GB/s for random 1-3 MiB reads, 8 in flight, depending on the moment. Every measurement below that reads the drive carries that spread.

How often experts repeat#

tools/moe_trace.cpp records each token's routed experts per layer, then replays them through an LRU cache per layer. The cache is warmed by the prompt, and the hit rate counts decode tokens only. Greedy decode, 300 tokens (Qwen3.6: 400), on Vulkan:

For Flash-Next, the per-token read from the drive and the ceiling that sets are:

These are ceilings from the drive alone, before compute, and from two prompts. The next sections add traces, compare policies and measure the real cache.

More traces: three families, three kinds of use#

tools/moe_trace.cpp --chat runs a conversation through the model's chat template and records every token's routed experts per layer. Prompt and decoded tokens are marked separately. All traces use greedy decoding on Vulkan with the engine's llama.cpp pin (2026-09-25), in three shapes: - code: one coding request, then 400 tokens; - chat: six user turns on unrelated topics, up to 256 reply tokens each; - long: about 30,000 tokens of the engine's own docs, a question, then 400 tokens.

tools/moe_policy.cpp replays a trace through several cache policies. Prompt tokens warm the cache, and only decode tokens count. - lru, lfu, slru: one cache per layer (slru is segmented: 20% probation). - g-lru, g-lfu: one budget shared by all layers (g-lfu halves its counts every 64 tokens). - opt: Belady's policy, which evicts the expert used furthest in the future. It needs the future, so it is the bound.

Decode hit rate with a cache of a quarter and of half of each layer's experts:

What the traces show: - One budget shared across layers (g-lru) is the best simple policy, or within a point of it, on every trace. Some layers need more slots than others, and a shared budget gives them those slots. - LFU fails on chat. A new topic makes last topic's counts stale: Coder chat at 32 per layer is 73.3% against 85.2% for LRU, and Qwen3.6 chat at 64 is 66.1% against 79.9%. Frequency with decay (g-lfu) recovers most of it, and is marginally the best on long context and at half-size caches. - Chat and long context repeat experts more than a single request. Coder at 32 per layer: code 68%, long 80%, chat 85%. - The bound is 5-16 points higher at quarter-size caches, so a smarter policy still has room. The traces are kept, so any policy can be tried on the same data.

Classic policies don't close the gap#

Three scan- and frequency-aware policies from storage caching, each over the shared budget (tools/moe_policy.cpp): ARC (adapts between a recency and a frequency list, with ghost lists), LRU-2 (evicts the expert whose second-to-last use is oldest), and W-TinyLFU as in Caffeine (a 1% or 10% LRU window; an expert leaving it enters the main cache only if it was used more often than the main cache's victim). Decode hit rate:

Across all 12 traces at 16, 32 and 64 per layer, none is more than 2.4 points above the shared LRU, and W-TinyLFU is up to 13 points below it (Qwen3.6 chat at 64). Routing has little popularity beyond recency for a frequency filter to use; the missing 7-22 points are in the future routes, which only a predictor can see.

Predicting the next layer's experts#

A trace also records a gate-ahead prediction for every decoded token. It applies layer l's router (with layer l's FFN norm, and GLM's selection bias) to the output of layer l-1: the engine has that vector while layer l's attention runs. The two-ahead prediction does the same from layer l-2's output, one layer earlier.

Predicting 2 x top-k covers 93-98% of misses, but reads 5-10 times as many experts that are never used. Predictors that need only the trace (the previous token's experts, co-occurrence with the previous layer) cover under 30%.

The expert cache in the engine#

moe/ holds the cache (Linux only, built into 1bit as ONEBIT_MOE): - gguf_index reads a GGUF header, including every shard of a split file, and finds each routed expert's bytes: blk.N.ffn_{gate,up,down}_exps.weight, one slice per expert. - expert_cache keeps experts in pinned slots. One mmap region is locked with mlock. Each list of slots (one per layer, or one per size class) keeps one packed array per part: part k of slot i is at part_off[k] + i * part_bytes[k]. That is the layout of a ggml expert tensor with as many experts as slots, so a GPU can compute on the slots directly. O_DIRECT reads land in each reader thread's aligned bounce buffer and are copied into place. - Misses are read with O_DIRECT, straight from the drive into the slot, by a pool of reader threads, so several reads are in flight. Demand reads go ahead of prefetches. - acquire(layer, experts) blocks until the experts are resident and pins them until release. prefetch(layer, experts) only queues reads. - Eviction is LRU, over one budget shared by all layers (the replays favour it) or per layer. - Slots come in size classes. Unsloth's UD quants use bigger types for some layers' experts, so layers are grouped by the byte sizes of their experts' parts, and each group gets its own slots (same share of experts per layer). Flash-Next's 9,216 slots pinned 34.4 GiB with one slot size and 27.0 GiB with size classes. Packing the parts brings Coder-30B's 1,536 slots from 4.4 to 4.1 GiB. - 1bit moe-cache replays a trace through the cache against the real model file, with real reads. Compute is simulated per layer: 1. at the layer's start, the prefetch is issued (lookahead 1: this layer's gate-ahead prediction; lookahead 2: the next layer's two-ahead prediction); 2. attention runs (--attn-frac of the layer's compute time); 3. acquire loads the routed experts, stalling on any read still in flight; 4. the experts compute (the rest of the layer's time).

It prints the decode rate the drive and cache allow, the stall per token, hits, prefetch hits, demand misses, wasted prefetches and bytes read.

What the cache delivers#

1bit moe-cache sweep, 2026-09-25/26: - Traces: all nine, 120 decode tokens each, the cache warmed by the last 64 prompt tokens. - Compute per token: the model's Vulkan0 decode rate with every expert resident. That is 12.1 ms for Coder-30B, 21 ms for Qwen3.6-35B and 14.1 ms for GLM, split half before and half after the experts are needed. - Box: Strix Halo was shared, with a load average of 6-9, other tenants' GPU servers, and 18 GB of their memory swapped out. A run can only be slowed by that, so the table takes the better of two runs. - Noise: small caches still vary by up to about 30% between runs, so the table shows the operating point that matters, 75% of the experts in RAM. - Drive ceiling: moe_policy computes it from the trace alone, with the shared LRU's misses at 3.7 GB/s. It is free of noise, and it covers the whole trace, so the cache is warmer than in the sweep's 120 tokens.

What the sweep shows: - Prefetch pays at a large cache. Two layers ahead is the best mode, or ties it, on 8 of 9 traces: 5-50% over no prefetch. It turns demand misses (2.6-6.2%) into prefetch hits and leaves 0.7-2.5% of reads on the critical path. It reads 10-50% more bytes, so it only pays when the drive has slack. - Prefetch hurts at small caches. At 25-50% of experts in RAM, the drive is the limit. Prefetch adds bytes (10-30% of its reads are wasted) and pushes out experts that were still needed. Two layers ahead predicts worse but gives a whole layer of lead, while one ahead gives only half of a layer's compute, about 0.1 ms: less than one 3 MiB read (0.9 ms). - The cache decides the speed. Coder-30B chat goes from 15-18 tok/s with 25% of experts (4.4 GiB) to 45-48 with 75% (13.2 GiB). The all-resident rate (93 tok/s) needs all 17.3 GiB. - Reads come in 3-6 MiB units at 3-3.9 GB/s. Four to sixteen reads in flight make no measurable difference on this drive.

The three-way split, measured#

Each MoE layer runs its attention and shared part (dense), then its routed experts. Batch-1 decode is serial across layers, so a split only pays if the experts' device is faster than the dense device would be. On Strix Halo every device reads the same LPDDR5X, so the question is which one reads weights fastest.

The expert block alone. tools/moe_expert_bench.cpp times one layer's routed experts for one decode token: gate and up MUL_MAT_ID, SwiGLU, then down. It uses the model's real layer-10 weights and fresh random top-k experts every run (median of 40). The NPU row comes from the closed-source Qwen3.6-35B NPU route's on-device stage profile (2026-09-24). That route requantizes to q4_1, and its block includes the shared expert.

HRX0 runs Qwen MoE layers through fused router and FFN dispatches in a real model, so its standalone number understates it for Qwen. In whole models, HRX0 decodes Coder-30B at 90.5 tok/s against Vulkan0's 92.4.

Whole models, split both ways. llama-bench -fa 1 -n 64 -r 3, pin 96f6b89. The routed expert tensors go to a second device with -ot exps=<device>, and everything else stays on the first (-ts 1/0):

Every split loses to one GPU device. Each layer hands its activations to the other device and back, and the other device reads experts no faster than Vulkan0. The NPU reads experts at 39-45 GB/s (the 35B route's MoE waves run at 80-84% of its ~53-56 GB/s streaming peak). Vulkan0 reads them at 135-171 GB/s. So for batch-1 decode: - the experts belong on Vulkan0; - HRX0 is an alternative for the whole model, not for half of it; - the NPU is worth using for a second model running at the same time, not for half of this one. On 2026-09-24, once both were loaded, the 35B NPU route decoded at 11.0 tok/s (11.1 solo) while a Vulkan decode ran at about 50. That is roughly 61 tok/s across two independent streams, with no visible interference.

Dense on HRX0 with the experts elsewhere fails for Qwen MoE models: HRX's fused Qwen router assumes the experts are on HRX0 too (#108). GLM, whose sigmoid router is not fused, runs.

The sweet spot per model#

Four axes: quantization (quality against speed), split, cache size (the RAM given to experts; the rest streams from the drive) and MTP. - Quality: KL divergence of each quant's token distribution from Q8_0's, and how often its top token matches, over 40 chunks of 512 tokens (Unsloth's UD quants, measured 2026-09-24). - Speed: Vulkan0 decode (tg64) and prompt (pp512), llama-bench -fa 1 -r 3. - Quants that are not on the box (the drive has 18 GB free) show sizes and quality only. Decode speed does not follow file size closely enough to estimate it: Qwen3.6-35B's Q8_0 is 39% larger than its UD-Q5_K_XL, yet only 5-10% slower.

Qwen3-Coder-30B-A3B (48 layers, 128 experts, top-8)

Qwen3.6-35B-A3B (40 layers, 30 DeltaNet + 10 attention, 256 experts, top-8)

GLM-4.7-Flash (47 layers, MLA, 64 experts, top-4 + shared): Q4_K_M only, 71.0-71.1 tok/s decode on Vulkan0 and 22.1-22.4 on HRX0. No other quant is on the box, so there is no quality axis.

Split: all on Vulkan0 for every model (see the three-way split). HRX0 is the whole-model alternative for Coder-30B (90.5 tok/s).

MTP: Unsloth ships no MTP head for these three models. Qwen3.8-Flash-Next has one. With its UD-Q4_K_XL fully resident and the MTP head drafting 3 tokens, Vulkan decodes at 40-49 tok/s against 25-26 without (2026-09-24, on Unsloth's qwen4exp/mtp branch; engine #110 carries the upstream MTP fix). With MTP, each verification step routes several tokens at once, so a layer needs the union of their experts. The Flash-Next section below measures what that costs a streamed cache.

The picks#

On Strix Halo with the whole box, every model here fits in RAM, so the pick is the fastest quant whose quality is acceptable, all on Vulkan0. When the RAM must be shared (other models resident, as in a fleet), the cache sets the speed:

MTP is available only for Flash-Next, and it is worth it both ways: 1.6-1.9 times resident, and it leaves the drive's reads per token unchanged when streamed.

Qwen3.8-Flash-Next: the model that needs streaming#

Flash-Next UD-Q4_K_XL is 111 GB: 71.7 GiB of routed experts, 26.8 GiB of per-layer token embeddings and 4.9 GiB of everything else. It has 48 layers of 512 experts, 10 routed per token, at 3.06 MiB each. With the whole box to itself it fits (about 77 GiB resident, with the embeddings read lazily). On a box shared with other models it does not.

Traces: MOE_CPU_EXPS=1 keeps the experts in the memory-mapped file on the CPU, so tracing needs no free memory for them, with the dense part on Vulkan0 (llama.cpp with qwen4exp, e71b805). Its layers mix four 2560-wide residual streams (hyper-connections), so the gate-ahead predictor does not apply yet, and these traces carry routes only.

MTP costs the drive nothing extra. A verification step routes the draft window, n + 1 = 4 tokens, and advances by the accepted ones (2.2 assumed, from the measured 40-49 against 25-26 tok/s). Replayed through the shared LRU, the reads per generated token are the same with and without MTP, within 0.5%, at every cache size, because each token's experts are read once either way. MTP groups them into steps of 2.2 times as many reads, which keeps more reads in flight. So MTP lowers the compute per token and leaves the drive term unchanged.

Drive ceilings (shared LRU, 3.7 GB/s), compute 38.5 ms per token without MTP (26 tok/s resident) and 22 ms with it (45.5 tok/s):

Measured on the drive (1bit moe-cache, chat and long traces, 192 per layer = 9,216 slots: 27.5 GiB of experts in 34.4 GiB of pinned slots. At the time, every slot was sized for the largest layer's expert. Size classes (below) now pin 27.0 GiB for the same slots):

This is far below the ceilings above, for two reasons: - The drive was slower than its fresh-box figure. Random 1 MiB reads, 8 in flight, gave 3.5 GB/s at one point in the session and 1.2-3.2 GB/s later, with the drive at 53 °C (limit 82) and the box's load average at 6-9 from other tenants. Paired runs (raw reads, the cache, raw reads again) put the cache's read path at the drive's rate of the moment. The cache is not the bottleneck. - Reads on demand are latency-bound. A layer misses 1-2 of its 10 experts (3 reads of about 1 MiB each: gate, up, down). They go in flight together, then compute waits. At that depth each read pays the drive's latency under load. With the oracle, all of a token's reads are in flight together: 1.4-4.8 times faster, from the same bytes.

So the design point for Flash-Next is queue many layers' reads at once. That needs predictions several layers ahead: its gate-ahead predictor, then two- and three-ahead. MTP's bigger steps help the same way, and a quieter drive helps most.

Running these measurements safely#

A model larger than free memory must never be loaded blind. llama.cpp's CPU backend copies quantized weights into anonymous memory (repacking), so n_gpu_layers=0 doesn't stream from the file. On 2026-09-25 that OOM-killed other services on the box. On Strix Halo, GPU allocations also escape cgroup memory limits. So every big run goes through scripts/mem-guard.sh <floor GB> <cmd>, which kills the run itself when MemAvailable falls below the floor.

Streaming in the inference path#

1bit serve --moe-slots N (docs/serve.md) streams the routed experts of each MoE layer from the model file while it decodes. The llama.cpp Vulkan pin (fork PR #19, 1bit/moe-stream) does the work, driven by environment variables that serve sets (ONEBIT_MOE_FILE, ONEBIT_MOE_SLOTS; src/llama-moe-stream.h lists the rest). The expert tensors load on the CPU memory-mapped (-ot exps=CPU --no-host --no-repack --load-mode mmap), so their pages are read only when a prompt batch touches them. Batches of up to 8 tokens (decode) use the slots; prompts keep the mapped path.

All three flags are needed. With -ot exps=CPU alone, llama.cpp puts the experts in a pinned Vulkan host buffer (Coder-30B: 16.9 GiB of RAM); --no-host alone gets a CPU repack (12.7 GiB); and this build's default load mode reads the file into memory. With all three, the experts are a CPU_Mapped buffer and the process holds 259 MiB of anonymous memory. 1bit serve --moe-slots passes all three. Before 2026-09-26 it passed only -ot exps=CPU, so the streamed runs below also held a full RAM copy of the experts. Decode never used that copy, so the speeds hold, but the memory saving did not.

• GPU mode (the default): each layer's slots live in a Vulkan device buffer the host maps, packed as expert tensors. A CPU op pins the batch's experts, reads the misses with O_DIRECT, and turns their ids into slot indices; the regular MUL_MAT_ID then runs on the GPU over the slots. A first version imported pinned host memory instead (buffer_from_host_ptr); the GPU reads that at about 1 GB/s, and it decoded 0.1-0.3 tok/s.
• CPU mode (ONEBIT_MOE_DEVICE=cpu): three custom ops (pin, gate/up/SwiGLU, down) on the CPU's dot kernels.
• Correctness (Qwen3-Coder-30B Q4_K_M, perplexity over 2 x 512 tokens): GPU mode gives exactly the all-Vulkan numbers, 21.1029 with 1-token batches and 20.9475 with 4-token batches.

Decode speed, Qwen3-Coder-30B-A3B Q4_K_M (48 layers x 128 experts = 6,144; llama-bench -n 64, 5 repetitions, warm ones; the first repetition starts from empty slots):

Through 1bit serve (a 128-token chat answer, three requests): 18.9, then 28.3 and 27.1 tok/s at 4,608 slots; 7.1, 6.6 and 6.0 at 1,536. The answers match the resident model's. With the prefetch moved off the decode path (below), 4,608 slots gives 17.1, then 38.9 and 39.0.

A model that fits in memory decodes fastest resident. Streaming is for the ones that don't: at a quarter of the experts in memory, Coder-30B still decodes 8-10 tok/s.

Where streamed decode spends its time#

ONEBIT_MOE_STATS=1 prints, at exit, the time spent in the remap op, the part of it spent waiting for reads, and the time between one layer's remap and the next (the GPU's work plus the round trip to the host). Coder-30B, -n 64 -r 4, per MoE layer (the numbers include the cold first repetition):

• The round trip caps streamed decode. 0.37 ms between remaps x 48 layers is 17.6 ms per token, about 57 tok/s, against 12.5 ms (80 tok/s) resident. Each layer's routed ids go to the host and its slot indices come back before the expert matmuls can run.
• Prefetch now pays a little. Its router scoring first ran inside the remap op, scalar, at about 0.15 ms per layer: prefetch cost 4,608 slots a third of its speed (28 against 42 tok/s). A worker thread now scores the next ONEBIT_MOE_PREFETCH layers (default 1) off the decode path.
• At 1,536 slots the drive is the limit. About 88 misses x 2.1 MB per token is 185 MB, and at 10 tok/s that is about 2 GB/s through O_DIRECT on a drive shared with other tenants. More reads in flight (ONEBIT_MOE_IO 4 to 32) didn't help, and prefetch trades demand misses for wasted reads (80 against 61 GiB over the run). What helps here is reading fewer bytes: a better hit rate (item 2 below) or smaller experts.

Routing that prefers resident experts#

Since no eviction policy recovers the misses, the other lever is the router itself. With ONEBIT_MOE_SUBST=r (1bit serve --moe-subst r), a streamed layer picks its experts with a CPU op in place of the top-k: when a chosen expert is not resident, the best resident expert among the router's next k choices takes its place, if it scores at least r times as high. The routing weights are then the substitute's own. The op also pins the chosen experts and writes their slot indices, in place of the remap op, so a layer still syncs with the host once (98 graph splits for Coder-30B with or without it; a first version with separate ops had 194).

Qwen3-Coder-30B Q4_K_M, KL divergence against the all-Vulkan logits (llama-perplexity --kl-divergence, 2 x 512 tokens, batch 1, prefetch on):

• r = 0.5 is the setting. Below it, reads hardly fall further while the divergence doubles at each step.
• What's left is prefetch. At 1,536 slots, demand misses fall from 6.8% to 1.5% of the routed experts. Most of the remaining reads are the gate-ahead prefetch's, and a prefetched expert counts as resident.
• Decode (llama-bench -n 64, warm repetitions, off and 0.5 interleaved over two rounds): 1,536 slots 8.7-11.9 tok/s against 7.9-10.1, and 3,072 slots 18.9-26.0 against 16.1-21.4. On a slower moment of the shared drive, 1,536 slots gave 8.0-12.4 against 6.0-8.7: the slower the drive, the more it gains (10-35%). At 4,608 slots, it makes no difference.

Flash-Next, streamed#

Qwen3.8-Flash-Next UD-Q4_K_XL (111 GB: 71.7 GiB of routed experts, 24,576 of them; 26.8 GiB of per-layer token embeddings) runs with 4.6 GiB on the GPU plus the expert slots. The experts and the per-layer embedding table stay file-backed (-ot exps=CPU,per_layer_token_embd=CPU --no-host --no-repack --load-mode mmap, which 1bit serve --moe-slots passes). The gate-ahead prefetch is off: its predictor does not fit this model's hyper-connections (Next, item 3).

Decode is timed as a batch-1 llama-perplexity pass over 512 tokens (every step a decode step). KL divergence is against the exact streamed run, which equals the model, since streaming is bit-exact. Exact and substituted runs were interleaved on the shared box:

(Reads include filling the slots from cold: about 13 GiB at 4,608 slots, 27 GiB at 9,216.) Perplexity of the exact runs: 11.32.

• A model larger than the box's free memory decodes at 3.5-6 tok/s exactly, from 18-32 GiB of GPU memory in all, instead of the 77 GiB it needs resident (40-49 tok/s with MTP).
• Substitution costs far more here than on Coder-30B. At r 0.5 the KL divergence is 0.07 (Coder: 0.008-0.012): a token spreads over 10 of 512 experts, and a stand-in differs more from the expert it replaces. r 0.9 keeps it at 0.015-0.028 for a 5-15% gain. Use r 0.9 or none on this model.
• The drive is the limit. At 4,608 slots, 4.0 ms per layer goes to waiting for reads against 1.7 ms of everything else.

Prefetch makes it worse. On Flash-Next, the gate-ahead predictor (router l+1 applied to layer l's 2560-wide FFN input) is right about two times in three, but every wrong guess costs a read from a drive that is already the limit. At 4,608 slots, depth 1 took 247 s per pass against 133 s without (240 against 157 GiB read), and depth 2 took 253 s. Keep ONEBIT_MOE_PREFETCH=0 on this model.

Reads in flight matter more than prefetch. ONEBIT_MOE_STATS times every read. With whole expert parts (about 1 MiB) and 8 reader threads, a layer's misses make only 6-7 reads, and each 1 MiB pread took 4.5-4.8 ms: about 1.8 GB/s, from a drive that gives 3.4 GB/s at that depth when the box is idle (rdtest: 3.2-4.1 GB/s idle, 1.3-2.9 under the other tenants' load). Reading parts in 256 KiB chunks with 16 threads keeps the queue deep (the streamer's default since fork PR #23; ONEBIT_MOE_CHUNK_KB, ONEBIT_MOE_IO):

That is 14-18% faster, with perplexity unchanged in all 8 runs. The rounds ran at load 7-22, which is why the baseline here is slower than the 133-145 s above. On Coder-30B at 1,536 slots the change is within the noise (four rounds, 6.7 against 7.0 tok/s).

Next#

0. Prompts through the slots. A batch above ONEBIT_MOE_MAX_BATCH (8) runs its experts on the CPU from the mapped file: Flash-Next prompts went at 0.4-0.8 tok/s through serve. Candidates: cap llama-server's ubatch at the streamed batch size, or stream larger batches when the layer's slots hold all their experts. (Measuring it was interrupted: the Flash-Next files were removed from the box on 2026-09-26.)
1. Fewer host round trips. The remap per layer caps streamed decode at about 57 tok/s on Coder-30B ("Where streamed decode spends its time"). 6,144 slots (every expert) runs erratically on the shared box.
2. A better stand-in than the router's next choice. On Flash-Next, r 0.5 costs a KL divergence of 0.07 ("Flash-Next, streamed"). Candidates: weighing a stand-in by how often it co-occurs with the missing expert, or reading only the missing expert's most-weighted rows.
3. Flash-Next's gate-ahead prediction. Its layers keep four 2560-wide residual streams (hyper-connections), so the predictor needs that model's mixing step before the router.
4. The quants not on the box (UD-Q2/Q3/Q4_K_XL): their decode speed, once there is room on the drive.

View this page's source on GitHub ↗