# BFCL tool-calling: TQ2Q vs same-size community IQ2_XXS — single-turn ledger

Qwen3.6-35B-A3B, our 4-level 2-bit (TQ2Q, 10.85 GB) vs bartowski IQ2_XXS
(10.68 GB). BFCL v4 @ 6ea5797, FC mode, temp 0, CPU-only (verified NEON path),
identical custom Qwen3.6 handler + prompts. Paired items; Wilson 95% CI;
McNemar exact two-sided (b = IQ2-right/TQ2Q-wrong, c = TQ2Q-right/IQ2-wrong).

| category | axis | n | IQ2_XXS | TQ2Q | Δ | McNemar |
|---|---|---|---|---|---|---|
| parallel_multiple | should-call | 200 | 76.0% [70,81] | **83.5%** [78,88] | **+7.5** | 14/29, p=0.032 |
| live_multiple | should-call | 200 | 76.0% [70,81] | 78.5% [72,84] | +2.5 | 12/17, p=0.46 (ns) |
| live_relevance | should-call | 16 | 87.5% [64,97] | 93.8% [72,99] | +6.2 | 0/1 (ns) |
| **irrelevance** | should-ABSTAIN | 240 | **88.3%** [84,92] | 39.2% [33,45] | **−49.2** | **118/0, p=6e−36** |
| composite | — | 656 | 80.8% | 66.0% | −14.8 | |
| should-CALL only | (excl. irrelevance) | 416 | 76.4% | **81.5%** | **+5.0** | |

## The finding (novel — no published 2-bit function-calling numbers exist)

**The 4-level quantization biases the model toward acting, not away from
competence.** When a tool call IS warranted, TQ2Q is *better* than the same-size
community 2-bit quant (+5.0 composite on should-call; +7.5 on parallel calls,
significant at p=0.032). It only fails at *abstention*: on tool-less questions
it fires a (well-formed but wrong) call 61% of the time vs IQ2's 12%.

The abstention damage is **purely one-directional**: McNemar 118/0 — of 118
discordant irrelevance items, *every single one* is IQ2-correct/TQ2Q-wrong,
zero the other way (p=6e−36). This is a systematic bias shift, not noise.

**Distributional metrics did not predict this.** TQ2Q is *better* than IQ2_XXS
on KLD (0.213 vs 0.246) and top-1 (81.4 vs 79.2), yet 49 pts worse at
"know when NOT to call." 2-bit PTQ damage here concentrates in conditional
instruction-following ("only call if applicable"), invisible to KLD/ppl/top-1.
First distributional-vs-behavioral inversion measured in this project.

## Deployment caveat (report honestly)

This is the *raw* model ability: BFCL's local path regex-parses raw completions,
so no grammar net engages and the model's own "should I call?" judgment is
measured directly. Production agent stacks that grammar-constrain and add
explicit abstention affordances would mask part of the irrelevance gap — but
the *bias direction* (eager to call) is a real property of the artifact and
would still surface as spurious calls on ambiguous inputs.

## Multi-turn — EARLY paired signal (80 of 400 items, not yet significant)

`multi_turn_base` only — the 80 items IQ2 had already finished, matched by a Mac
TQ2Q run (BFCL v4 multi-turn: stateful execution + end-state match; same
CPU/NEON path, custom Qwen3.6 handler, temp 0). This is a **preview**; the full
400-item multi-turn set (multi_turn_base 80–199 + 200 multi_turn_long_context)
is staged for the c8g Graviton box. Read the CIs, not the point estimates.

| category | axis | n | IQ2_XXS | TQ2Q | Δ | McNemar |
|---|---|---|---|---|---|---|
| multi_turn_base (subset) | task-complete | 80 | 42.5% [32,53] | 33.8% [24,45] | −8.8 | 17/10, p=0.25 (ns) |

**Read:** on multi-turn task completion TQ2Q trails IQ2 by ~9 pts on this subset,
but the gap is **not significant** (McNemar b/c 17/10, p=0.25; CIs overlap
heavily). Directionally this is consistent with the single-turn story — 2-bit
PTQ damage concentrating in multi-step instruction-following rather than raw
capability — but n=80 cannot confirm it. **Do not draw a multi-turn conclusion
until the full 400-item c8g run lands** and paired_analysis is re-run over all
of multi_turn_base + multi_turn_long_context.

## Reasoning-ON re-eval (GPU) — the abstention gap is mostly a no-reasoning artifact

The rows above ran with the model's **reasoning DISABLED** (the BFCL handler forced
`enable_thinking=false` for both legs). Re-ran the same 656 paired items through the
same harness with **only that flag flipped ON**, on the **Apple GPU** (using the new
TQ2_0 Metal kernel — decode path). This is a *distinct regime* from the rows above, not
a replacement (GPU greedy ≠ CPU exactly on the 35B; reasoning changes generation) —
read the off→on deltas.

| category | axis | n | IQ2 off→on | TQ2Q off→on | Δ on (TQ2Q−IQ2) | McNemar (on) |
|---|---|---|---|---|---|---|
| **irrelevance** | should-ABSTAIN | 240 | 88.3→83.3 | 39.2→**76.7** | **−6.7** | 24/8, **p=0.007** |
| parallel_multiple | should-call | 200 | 76.0→80.0 | 83.5→81.5 | +1.5 | 22/25, p=0.77 (ns) |
| live_multiple | should-call | 200 | 76.0→78.0 | 78.5→76.5 | −1.5 | 17/14, p=0.72 (ns) |
| live_relevance | should-call | 16 | 87.5→87.5 | 93.8→93.8 | +6.2 | 0/1, p=1.0 (ns) |
| composite | — | 656 | 80.8→80.8 | 66.0→**78.5** | −2.3 | |
| should-CALL only | excl. irrelevance | 416 | 76.4→79.3 | 81.5→79.6 | +0.3 | |

**The headline finding needed revising.** The "catastrophic abstention failure" was
overwhelmingly a **no-reasoning eval artifact**:
- **Abstention gap collapses from 49.1 pts → 6.7 pts.** TQ2Q's irrelevance abstention
  nearly doubles (39.2 → 76.7) once it can reason "this tool doesn't fit" before deciding.
  The reasoning-off gap was McNemar 118/0, p=6e−36 (every discordant item TQ2Q-wrong);
  reasoning-on it's 24/8, p=0.007 — **still significant, but small**, not catastrophic.
- **IQ2 slips slightly** (88.3 → 83.3): it was already cautious without reasoning, and CoT
  nudged it toward a few more spurious calls. So the gap closes from *both* sides.
- **TQ2Q's should-call edge evaporates to parity.** Its reasoning-off advantage
  (+5.0 composite should-call, +7.5 on parallel) disappears (Δ +0.3 should-call, ns) —
  reasoning lifted IQ2 more on should-call than it did TQ2Q.
- **Composite gap −14.8 → −2.3.** With native reasoning enabled, 2-bit TQ2Q is at
  **near-parity** with the same-size community quant across all axes; a small (6.7 pt,
  significant) abstention deficit is what remains of the "eager-to-call" story.

**Takeaway:** the eager-to-call bias is real but **regime-specific** — it shows up when the
model is forced to answer without reasoning. Any deployment that lets the model reason
(the default for this model) recovers most of the abstention. The distributional-vs-
behavioral inversion still holds (KLD didn't predict the reasoning-off gap), but the
*magnitude* of the behavioral penalty was an artifact of the no-reasoning eval config.