Reproduce: TQ2Q speed across GPU backends

Everything below uses public artifacts and a fixed llama-bench protocol. No special hardware or private builds — clone, build with your GPU backend, download the public GGUFs, run.

1. Build the kernels

The TQ2_0 kernels (CUDA / Metal / HIP / Vulkan) live on our fork's master:

git clone https://github.com/AgntroAI/llama.cpp
cd llama.cpp
# pick the backend for your machine:
cmake -B build -DGGML_CUDA=ON  -DCMAKE_CUDA_ARCHITECTURES=<your arch, e.g. 86>   # NVIDIA
#   -DGGML_METAL=ON     (Apple Silicon)
#   -DGGML_HIP=ON       (AMD ROCm)
#   -DGGML_VULKAN=ON    (any Vulkan device, incl. Intel/AMD iGPUs)
cmake --build build --target llama-bench -j

2. Get the models (public, anonymous download)

• 35B — Agntro/qwen3.6-35b-a3b-TQ2Q-GGUF (~11 GB)
• OLMoE-1B-7B — Agntro/olmoe-1b-7b-TQ2Q-GGUF (~2 GB; fits an 8 GB card)

huggingface-cli download Agntro/olmoe-1b-7b-TQ2Q-GGUF --local-dir olmoe

3. Run the benchmark (identical everywhere)

llama-bench -m <model.gguf> -ngl 99 -p 512 -n 128 -r 5     # full GPU offload
llama-bench -m <model.gguf> -ngl 0   -p 512 -n 128 -r 5     # CPU baseline for the speedup

Report pp512 (prefill t/s) and tg128 (decode t/s), median ± stddev. -r 5 for stability.

• llama-bench does not accept --no-repack (it errors); full -ngl 99 offload doesn't use the CPU repack path anyway. If a model doesn't fit VRAM, drop to the largest -ngl N that fits and record N (that's the honest "big model on a small card" hybrid number).
• Verify correctness on a new backend before trusting a bench (perplexity ≈ the CPU baseline).

Hazard