Our 2-bit MoE runs on the GPU — the kernel just isn't merged yet

We measured our 35B 2-bit mixture-of-experts model (TQ2Q) on an NVIDIA A10G expecting to confirm it "runs on the GPU." The full arc turned out to be more interesting: on mainline llama.cpp there is no CUDA kernel for the 2-bit TQ2_0 format, so the experts compute on the CPU — yet full offload still makes decode 4.1× faster, because everything except the tiny 2-bit experts moves to the GPU. And when we built the real kernel — compilade's unmerged TQ2_0 CUDA branch — the format proved genuinely GPU-native: experts move into VRAM, prefill jumps ~11×, and our fourth code level survives on the GPU bit-for-bit. The speedup is real today as hybrid offload; it becomes native the day the kernel lands.

What we expected — and why it was wrong

TQ2Q packs Mixture-of-Experts weights into llama.cpp's 2-bit TQ2_0 container, but uses a fourth code word (q=3 → +2×scale) that plain "ternary" leaves empty. We wanted to know whether a GPU would read that fourth level correctly or strip it. So we ran each GGUF CPU-only (-ngl 0) versus fully offloaded (-ngl 99) and compared perplexity on WikiText-2, reasoning: if q=3 is stripped on the GPU, perplexity collapses; if it matches, q=3 survived.

The perplexity matched — OLMoE bit-identical, the 35B within 0.1%. We read that as "the fourth level survives on CUDA." That inference was invalid, for a subtle reason: TQ2_0's dot product is integer-exact. A quaternary weight times an int8-quantized activation is a deterministic integer sum, so CPU and GPU produce the same bits regardless of which device runs it. A bit-identical match is therefore not evidence the experts ran on the GPU — nor evidence they didn't. (We later confirmed this the hard way: a build with the experts provably in VRAM is also bit-identical.) Perplexity simply isn't a placement discriminator for this format.

So we used the evidence that actually discriminates. First, VRAM: at -ngl 99 on mainline, the OLMoE process holds only 801 MiB of GPU memory — the experts are sitting in host RAM, not VRAM. Second, a matmul-path sweep (Experiment B): forcing the cuBLAS path (LLAMA_FORCE_CUBLAS), forcing MMQ, and the default all produce the same speed. If the experts were computing on the GPU, those paths would diverge; that they don't means no CUDA matmul is running for them. Both point the same way: mainline llama.cpp has no CUDA kernel for TQ2_0 — grep ggml-cuda/ and you get nothing — so the ggml scheduler keeps every 2-bit expert matmul on the CPU backend even at -ngl 99. The 35B's 0.1% perplexity wobble is just rounding on the offloaded non-expert tensors (attention, shared expert, head) that did move to the GPU; it says nothing about the experts either way.

So why is "the GPU" 4× faster?

Because -ngl 99 doesn't move the experts — it moves everything else. Per token, our 35B reads ~1.97 GB of weights, and only ~260 MB of that — about 13% — is the TQ2_0 routed experts. The other ~1.7 GB is not 2-bit: F16 attention, the Q5_K always-on shared expert, the Q6_K vocab head, the router. Full offload puts that 1.7 GB on the GPU's ~600 GB/s while the small 2-bit slice rides on the CPU. Decode is memory-bound, so throughput follows the bytes:

Decode: hybrid CPU+GPU

The 35B jumps from 14 to ~59 tokens/sec — not because the 2-bit weights are on the GPU (they aren't), but because the 87% of per-token bytes that aren't 2-bit are. The A10G shares the RTX 3070 Ti's memory bandwidth, so a consumer card with room for the non-expert tensors would do much the same.

Prefill barely moves

Prefill — digesting the prompt before the first token — is a different story, and now a predictable one:

The GPU is barely ahead of the CPU here — 1.07×, and flat from 512 to 2048 tokens. Which is exactly what you'd expect on mainline: prefill is compute-bound, and the expert matmuls — the bulk of that compute — run on the CPU regardless of -ngl. Offloading everything else can't move a number the CPU still gates. Hold onto that "flat 1.07×": it's a property of the mainline build with the experts on the CPU, not a property of the format. The next section is what happens when the experts are allowed onto the GPU.

The kernel exists — someone built it

Mainline has no merged TQ2_0 CUDA kernel, but that isn't the same as "no kernel exists." compilade's PR #11183 (compilade/cuda-tq2_0) is a real, working TQ2_0 MMQ kernel — it's just open and unmerged. So we built the branch and ran the measurement mainline can't: experts on the GPU.

They move. At -ngl 99 the OLMoE process now holds ~2 GB of VRAM instead of mainline's 801 MiB — that ~1.2 GB is the 2-bit experts, in VRAM, where mainline left them in RAM. And with the experts finally on the GPU, prefill unflattens:

Three things fall out of the real kernel. Prefill goes ~11× (333 → 3737 t/s at pp2048) — and it now rises with the micro-batch occupancy sweep instead of sitting flat, the fingerprint of a compute path that scales with GPU work. The mainline "flat 1.07×" was the artifact; this is the format. The matmul path now matters: where mainline's cuBLAS/MMQ/default were indistinguishable (~335 t/s — the tell that nothing was running on the GPU), on #11183 they separate (cuBLAS 2461 < default 2603 ≈ MMQ 2614), exactly as a real CUDA matmul should. And our fourth level survives on the GPU: running q=3 through the genuine TQ2_0 MMQ kernel gives KLD 0.3007 vs the shipped CPU model's 0.2999 — a match. The quaternary code isn't a CPU-only trick; it's correct on CUDA.

Why it matters

There are two takeaways, and both are useful. The first is for today: you don't need a merged GPU kernel to get a GPU speedup. A 2-bit MoE puts almost all of its parameters into experts that are individually tiny and only sparsely activated — so the parts actually worth accelerating (attention, the always-on shared expert, the head) are the unquantized parts, and those already have GPU kernels. Offload them, leave the 2-bit experts on the CPU, and get ~4× decode on a single 24 GB card straight off a stock llama.cpp build. It's hybrid, not native — but it's a real lever for anyone with a mid-range GPU and a CPU with bandwidth to spare.

The second is for when the kernel lands: TQ2Q is not a CPU format that happens to offload well — it's GPU-native. The real TQ2_0 CUDA kernel exists, moves the experts into VRAM, takes prefill to ~11×, and reads our fourth level correctly. The only thing between mainline users and the native path is a merge (or a git checkout of the branch and a build).

Two honest limits. compilade's branch is based on a January-2025 llama.cpp, so only prefill is a clean cross-build comparison — a decode delta between mainline and the branch would be codebase age, not expert placement, so we don't report one. And the kernel is unmerged: mainline users get the 4× hybrid today; the ~11× prefill and native q=3 arrive when it lands upstream, or immediately if you build the branch yourself.

Hybrid figures measured with llama-bench, single stream, on mainline llama.cpp master (build 9892, July 2026 — the TQ2_0 CUDA kernel, PR #11183, was still unmerged, so ggml-cuda/ is grep-clean for it) on an AWS A10G. The GPU-native figures are from a build of compilade/cuda-tq2_0 (PR #11183, fbddb262), where the experts run in VRAM; because that branch predates current master, only prefill is compared cross-build. Corrected 2026-07-07: perplexity does not discriminate tensor placement for an integer-exact format — VRAM footprint and the matmul-path sweep do.