# Qwen3.6-35B-A3B TQ2Q — MMLU (Derek Colley's exact mmlu_llama config)

Run: 2026-07-08, RunPod RTX 3090, AgntroAI/llama.cpp merged kernels (TQ2_0 CUDA),
llama-server + lm-eval local-chat-completions, mmlu_llama 5-shot, /no_think,
gen max_tokens=512 temp=0.7 top_p=0.8 top_k=20, num_concurrent=4.

## Real score (re-scored from --log_samples; raw strict_match reads 0.0 — filter bug)
The lm-eval `strict_match` filter fails to extract the letter over chat-completions
(gen_prefix "The best answer is" isn't prefilled). The model answers cleanly in the
"The best answer is X" format EVERY time (0 unparsed / 14042). Regex re-score:

**OVERALL: 11270/14042 = 80.26%**
- STEM         77.55% (2445/3153)
- Humanities   74.77% (3518/4705)
- Social Sci   88.11% (2711/3077)
- Other        83.55% (2596/3107)

## Head-to-head (Derek's leaderboard, same config)
- Qwen3 reference (full precision): ~84.76%
- Nemotron (full):                   80.79%
- **Ours, 2-bit TQ2Q (~10 GB):       80.26%**  → ties Nemotron, -4.5 vs Qwen3 ref.

Regex used: (?:the best answer is|answer is|answer:)\s*\(?([A-D])\)?  (case-insensitive)