The stance beats the model
We threw max reasoning at conceptual flaws — 21 runs across 7 models, zero catches. Then we changed the question. Same models, default effort, and the hardest flaw went from uncatchable to caught. The role is the lever, not the model.
The blind spot
In the safety-net probe, one bug was
missed by every model, in every role, at every effort level. CONCEPT-3: a
len(text)//4 tokenizer heuristic that's a flawed mental model.
It looks reasonable — the code runs, the approximation is widely cited — but
it's structurally wrong in a way that matters under load.
The conclusion there was "reviewers can't catch conceptual flaws; you need tests." We accepted that and built a contract-verifier. But it nagged: is the conceptual flaw class truly invisible to LLM review, or were we asking the wrong question?
The hypothesis
Reviewer, observer, adversary — all three safety-net roles share a stance: find a defect. They scan the output for things that are wrong. But a conceptual flaw isn't a defect you find by scanning. It's an assumption that's unsound. To catch it, you have to do a different cognitive operation: enumerate the assumptions, stress each one against the domain, and flag the unsound ones.
So we built a fourth role — the conceptual reviewer — with a different stance: not "find defects" but "list every implicit assumption, stress each against the domain, flag unsound ones + the fix." The prompt is general (no hardcoded flaws — that would be overfitting). The question: does the stance unlock conceptual-flaw detection, or do you just need a smarter model / more reasoning?
The controlled test
The role is the variable. Model and effort are controlled.
| Condition | CONCEPT-1 caught? | CONCEPT-2/3 caught? |
|---|---|---|
| observer/reviewer/adversary @ DEFAULT | 0/2 models (0%) | mixed |
| observer/reviewer/adversary @ MAX reasoning (7 models × 3 roles = 21 runs) |
0/7 (0%) | — |
| conceptual-reviewer @ DEFAULT | 2/7 (MiniMax-M3, mimo-v2.5-pro) | 7/7 universal |
21 runs at max reasoning: zero catches. Every model, every existing role, maxed effort. The "find a defect" frame is structurally blind to this flaw class — and no amount of reasoning fixes it.
Switch the stance, default effort: the flaw appears. The same two models (MiniMax, mimo) that missed CONCEPT-1 in the defect-search frame at max reasoning (0%) catch it in the assumption-enumeration frame at default effort. And CONCEPT-2 and CONCEPT-3 — two other conceptual flaws the existing roles missed — are caught by all 7 models in the conceptual-reviewer role.
Why this is decisive
The comparison holds model and effort constant. MiniMax-M3 and mimo-v2.5-pro are in both arms. At max reasoning in the defect-search role, they miss CONCEPT-1. At default effort in the assumption-enumeration role, they catch it. The only thing that changed is the question being asked.
Maxing reasoning didn't just fail to help — it was actively wasteful. 21 runs at max effort cost ~10–40× the tokens (plus truncation failures on some models), for zero marginal catches. The stance did what the compute couldn't.
The role generalizes
This isn't a one-flaw trick. CONCEPT-2 (a 32-char cache key that's too short to avoid collisions) and CONCEPT-3 (a success rate computed from 2 samples) are different flaws in different domains — and the conceptual reviewer caught them universally (7/7), while the existing three roles missed them at every effort. The assumption-enumeration stance catches a class of flaws, not a single example.
CONCEPT-1 specifically is hard even for the role (2/7) — likely because
len//4 is so widely cited as "fine" in training data that it reads
as canonical. MiniMax and mimo are the strongest conceptual reviewers by this
measure; they're willing to question the canonical.
What we learned about ourselves
Honest precision caveat. The conceptual reviewer was
rigorous enough to find real flaws in items we labeled "clean" —
e.g. int(raw) throws on non-numeric store values, which is
genuinely buggy. So some "false alarms" on the clean set were actually the
model being more careful than the test designer. The clean items need
tightening before precision is trustworthy. Recall is solid.
What this means
- The conceptual reviewer is a 4th role. It earns its place alongside reviewer/observer/adversary — it catches a flaw-class the others structurally miss. Adding it to the panel is free recall.
- Don't default to max reasoning for the safety net. 21 runs, 0 catches, 10–40× cost. Effort is per-call/per-model, set from a measured test-set, not a global dial.
- Model→role assignment matters. MiniMax-M3 + mimo-v2.5-pro lead as conceptual reviewers; GLM-5.2 led as reviewer/observer (from the earlier probe). No single model is best at every role — the assignment is measured, not assumed.
The bigger picture
There's a temptation to solve every LLM weakness with more compute: bigger model, more reasoning, longer context. This probe is evidence against that reflex. The conceptual-flaw blind spot wasn't a model-capacity problem — it was a question problem. The models had the knowledge; they were applying the wrong operation.
The safety net now has four roles, each with a different cognitive operation. Not because more is better — but because defect-search and assumption-enumeration are genuinely different tasks, and no amount of reasoning in one stance substitutes for running the other.