Does 2-bit break tool-calling — and how fast is it? TQ2Q across the GPU fleet

TQ2Q = a 4-level (quaternary) 2-bit quant of the routed MoE experts, packed into llama.cpp's stock TQ2_0 container by reclaiming its unused 4th code word. Until this month it ran only on CPU. It now runs GPU-native on all four backends — so we can finally answer both questions with real numbers.

2026-07-08 — answers the two questions @DerekColley_ asked: (1) does quaternary-PTQ hurt tool-calling reliability (the real agent bottleneck), and (2) per-token prefill-vs-decode latency, now that our TQ2_0 GPU kernels are merged. All numbers are single-stream llama-bench on our fork AgntroAI/llama.cpp master (bb0d956), which carries native TQ2_0 kernels for CUDA / Metal / HIP / Vulkan. Full protocol to reproduce: reproduce.md.

1. Tool-calling: does 2-bit break it? Largely no — if you keep reasoning on.

The honest agent question isn't tok/s, it's whether the quant still calls the right tool and, just as important, abstains when no tool applies. We ran BFCL v4 (@6ea5797), our Qwen3.6-35B-A3B TQ2Q vs the same-size community IQ2_XXS, official chat template, reasoning on:

Small deltas, none strongly significant. TQ2Q is competitive with the same-size 2-bit baseline.

The finding worth keeping: for a 2-bit model, abstention is a reasoning act. A thinking-off harness made TQ2Q look catastrophic — 39.2% abstention (McNemar 118/0, p≈6e-36, a systematic bias toward acting). Flip reasoning on and it recovers to ~77–80%; the gap collapses 10×. We measured this on two independent kernels (our CUDA run and the Mac's Metal run — 79.9% and 76.7%). So the deployment rule is concrete: don't suppress chain-of-thought for latency and your 2-bit model tool-calls fine. (Full paired ledger + raw JSONL — re-score it yourself.)

2. Prefill vs decode, across seven GPU configs

Same protocol everywhere: llama-bench -ngl 99 -p 512 -n 128 -r 5.

Three things this says:

1. Discrete VRAM (NVIDIA CUDA) wins on both axes. OLMoE full-offload is 4.7× decode on a consumer 3070 Ti. And the headline for the size-conscious: the 35B doesn't fit 8 GB, but with an expert-offload split (put the 2-bit experts on the GPU, -ot) it hits 85.8 t/s decode on an 8 GB gaming card — 5× its CPU, and 1.85× the naïve -ngl split. That's the real "run a 35B on the GPU you already own" story.
2. Unified-memory GPUs are prefill accelerators. Apple M4 and the Intel Arc 140V both show 2–6× prefill but decode ≈ wash — because decode is bandwidth-bound and the GPU shares the same LPDDR bus as the CPU. Metal and Vulkan independently show the identical shape. For agentic/RAG/code workloads (prefill-heavy), that's a real win; for long free-form generation, less so.
3. Weak iGPUs are an honest non-win. The Intel Iris Xe loses to 8-thread AVX2 (0.2× prefill) — a weak UMA iGPU doesn't beat a tuned CPU kernel. The Arc 140V (a much stronger iGPU) does win prefill. So "runs on your laptop iGPU" is true — but only on capable iGPUs, and as a prefill accelerator, not a decode one. We report the loss because it's real.

q=3 correctness was re-verified on every backend (KLD vs bf16 ≈ CPU baseline; e.g. Aorus CUDA OLMoE KLD 0.2994, Vulkan 0.3001).

3. Quality holds

Our 35B TQ2Q is a two-axis Pareto win vs the same-size community IQ2_XXS — better ppl (6.79 vs 7.02), KLD (0.213 vs 0.246), and top-1 (81.4 vs 79.2) and faster. The 80 GB-class giants run GPU-native too: DeepSeek-V4-Flash TQ2Q holds perplexity 6.16 on a single H200 (q=3 intact).

But distributional metrics (ppl/KLD) only go so far — the real question is task accuracy. So we ran the full mmlu_llama (Derek Colley's exact 5-shot /no_think config, all 14,042 questions) on the 2-bit 35B and put it next to the other models on his leaderboard:

¹ Derek's own mmlu_llama numbers. "Qwen3.6 base" is unsloth/Qwen3.6-35B-A3B-MTP-GGUF — itself a quantized unsloth GGUF (the repo spans 1-bit → bf16; the recommended llama.cpp build is the ~4-bit UD-Q4_K_XL), the un-fine-tuned base (vs. the AgentWorld fine-tune). So this is a 2-bit vs ~4-bit comparison of the same model, not 2-bit vs full precision.

So a 2-bit, ~10 GB model ties Nemotron (−0.5), beats the AgentWorld fine-tune (+1.5), and lands 4.5 points under the base Qwen3.6 — and that base is itself a ~4-bit quant at roughly twice our bytes. We even lead Nemotron on Social Sciences and Other. The gap concentrates where you'd expect 2-bit to hurt: Humanities (−3 vs Nemotron; moral_scenarios / professional_law heavy) and math-heavy STEM. That's the honest 2-bit fingerprint — recall and applied knowledge hold; multi-step reasoning is where the bits are missed.

4. The giants — and a runtime mystery we chased down (honestly)

The 80 GB-class giants are the part where you should not trust a marketing bullet, so here's the real engineering. On the H200 we hit something odd: DeepSeek-V4-Flash (deepseek4) decodes at ~30 t/s, while our similarly-sized Hy3 decodes at 86 t/s — same GPU, same TQ2_0 kernel, and V4F actually has fewer active params. A 2-bit format that's smaller and yet 3× slower makes no sense, so we dug in instead of shipping a number.

The answer: it's not the quant, it's the runtime. Perplexity (a forward pass over fixed tokens — same weights, same q=3 kernel) is clean at 6.16. A single-stream llama-bench shows V4F prefill is normal (1251 vs Hy3's 1318); only decode is off. And a standard-attention giant on the identical kernel decodes 3× faster. The tell is the architecture: V4-Flash uses DeepSeek Sparse Attention — a lightning indexer, compressed-KV streams, and hyper-connections — which today's llama.cpp expresses as hundreds of unfused ops with no dedicated sparse-decode kernel, where a stack like vLLM has a single fused one. That's a runtime-maturity gap, not a defect in the weights. (We confirmed it by reading both codebases; the prefill/decode split and the same-kernel Hy3 comparison nail it.)

So the honest V4-Flash story is: memory (80 GB vs ~280 GB fp16), prefill, and quality are real; generative decode is bottlenecked by the sparse-attention runtime. We put that caveat right on the model card — and we're fixing the runtime ourselves on our fleet rather than waiting.

And Hy3: our 83 GB 2-bit is ~24% smaller than the community whole-model Q2_K (109 GB) and decodes at 85.9 t/s GPU-native at perplexity 6.60 — a size-quality tradeoff, competitive, not a "win on both." An earlier internal comparison that looked like a loss turned out to be measured on an outdated CPU-hybrid path against a 30%-larger model; we corrected the framing rather than quietly drop it.

That's the point of this whole post: report the wins and the losses, and when a number looks wrong, find out why before you publish it.

Honest caveats (so the numbers mean something)

• Single-stream llama-bench, not batched-server throughput — comparable to a per-token latency ask.
• Lunar's 35B decode is thermally bounded. The Arc 140V is in a fanless-class Lunar Lake ultrabook; sustained 35B decode throttles hard (we saw tg swing 9–16 across runs). The iGPU runs it and wins prefill, but a passively-cooled laptop can't sustain 35B decode — reported from the coolest run.
• CPU baselines are per-machine, per-build (MinGW vs MSVC vs NEON differ); we compare within a box.
• The 35B "expert-split" number needs the -ot flag; naïve -ngl on an 8 GB card is slower.

Reproduce

Full protocol (build flags per backend + the exact llama-bench commands): reproduce.md. Models are public GGUFs (anonymous download): Agntro/qwen3.6-35b-a3b-TQ2Q-GGUF, Agntro/olmoe-1b-7b-TQ2Q-GGUF. Kernels: AgntroAI/llama.cpp master. Never re-quantize a TQ2Q GGUF (it strips the 4th level) — download and run.