The best coder was the weakest reviewer

Tencent's Hy3 (295B MoE) is the most efficient coder we've measured — 100% coverage, 28 calls average, zero cost. As a conceptual reviewer it was the weakest in the cohort at 50% recall. One model, opposite extremes — and exactly why role→model assignment must be measured, not assumed.

The setup

We benchmark models in isolation per role — the methodology behind our earlier safety-net probe and conceptual-reviewer finding. Each model gets tested as a reviewer (does it catch planted bugs?), an adversary (can it falsify a finding?), and a conceptual reviewer (does it spot unsound assumptions?). Ground truth is known — we planted the bugs.

Hy3 was new: a 295B mixture-of-experts model from Tencent, available free on OpenRouter for a promotional window. 256K context, tool-calling, and reasoning support. We ran it through four evaluations:

1. Safety-net probe — 8 planted-bug items × 3 roles (reviewer, observer, adversary). Measures recall and false-alarm rate.
2. Conceptual reviewer probe — 5 assumption-flaw items. The hardest review role.
3. Coder fidelity eval — implement a 15-symbol spec against the real FastAPI codebase. Measures plan-coverage and plan-discipline (drift).
4. End-to-end pipeline run — a full task through the harness with Hy3 as the coder.

The coder result

Coverage is saturated. Every model, in every run, implemented all (or nearly all) of the 15 specified symbols. This task — a well-specified leaf task from a plan — is within every frontier model's competence. The differentiator is efficiency: Hy3 averaged 28 calls, DeepSeek 44, MiniMax 74, GLM 67.

Discipline is the secondary axis: DeepSeek had zero drift across all 4 runs (100%). Hy3 added one extra symbol in one run (98%). MiniMax and GLM each added one extra (94–97%). All disciplined — none invented features or expanded scope.

The code quality was clean across the board. Real implementations, not stubs. Proper type hints (Sequence[str] | set[str] | None), docstrings, correct FastAPI types (BaseRoute, APIRoute), organized with section headers.

The reviewer result

Hy3 numbers are averaged across 4 runs (safety-net + conceptual). The other models are from our earlier safety-net probe (n=1).

Hy3 is a disciplined reviewer and observer — 100% precision, zero false alarms across all runs. Recall sits at 75%, missing subtle bugs. As an adversary, it's noisy: precision swings 60–100% across runs. As a conceptual reviewer, at 50% recall it's the weakest in the cohort but not dramatically so — it's 16 points below the 66% cluster, and catches the low-sample-flaw (CONCEPT-3) reliably.

One model, opposite extremes

Hy3 is the most efficient coder we've measured — 100% coverage across 4 runs, 28 calls average, near-zero drift, free. But it's the weakest conceptual reviewer in the cohort at 50% recall. It's not a "bad model." It's a model that's extremely good at one thing and weaker at a different thing, and the two things happen to be what a coding-agent pipeline splits into separate roles.

And cost doesn't change this. Hy3 is free — $0 per million tokens — and it's the most efficient coder. But "free and efficient at coding" doesn't mean "use it everywhere." A pipeline that used Hy3 as its conceptual reviewer would catch fewer assumption flaws than GLM or MiniMax. A pipeline that used it only as the coder and paired it with a stronger reviewer gets the best of both: free, high-fidelity implementation, disciplined review.

The free-tier catch

One practical finding: OpenRouter's free tier rate-limits at 16 requests per minute. The coder loop fires tool calls as fast as the model responds — without throttling, it hit 429 Rate Limit Exceeded after 35 calls. We added a rate-limiting wrapper (4.5s minimum interval between calls + exponential backoff on 429) and the run completed cleanly.

This shapes architecture: a free model isn't a drop-in replacement for a paid one if the pipeline assumes high concurrency.

What it would have cost

The free promotion won't last. The paid variant (tencent/hy3) is $0.20/$0.80 per million tokens (input/output) — among the cheapest models available. For context, here's what the full evaluation battery would have cost at paid rates:

Even at paid rates, Hy3 is the cheapest model in the comparison — nearly 5× cheaper than GLM-5.2 on TokenRouter, which is the most expensive:

Five cents for the entire evaluation battery. At nominal rates Hy3 is cheaper than DeepSeek — but there's a catch. DeepSeek's prompt-cache pricing is aggressive: cached tokens cost $0.003625/M (0.8% of input), while Hy3 charges $0.05/M (25% of input). Our pipeline achieves ~97% within-model cache hit at the coder fork (the A8/B2 finding), and at that rate DeepSeek-pro is actually 13% cheaper than Hy3. The break-even is at ~88% cache hit: below it, Hy3 wins; above it, DeepSeek's cache subsidy dominates. For workloads with low cache reuse (one-shot calls, cold starts), Hy3 is the cheaper choice.

What we wired

Based on these numbers, Hy3 is now configured in the harness as:

• Coder role: primary candidate. 100% coverage, most efficient, free.
• Reviewer/observer role: viable backup. 75% recall, 100% precision.
• Adversary role: no. Precision swings 60–100% across runs.
• Conceptual role: no. 50% recall — weakest in the cohort.

The recommended pairing: Hy3 as coder + GLM-5.2 as reviewer + MiniMax-M3 as adversary/conceptual. The free coder produces reliable, high-fidelity implementations; the disciplined reviewers catch what it can't self-assess. Total pipeline cost: just the reviewer calls.

Honest limitations

• Coverage saturation. All four models hit ~100% coverage on this task. The coder eval distinguishes efficiency and discipline, not raw capability — a well-specified leaf task is within every frontier model's reach. Harder tasks (multi-file integration, ambiguous specs) would likely separate the cohort more.
• Variable n. DeepSeek and Hy3 have n=4; MiniMax and GLM have n=2 (TokenRouter rate-limits sustained calls). The reviewer probe ran 4 times for Hy3; precision is rock-stable, recall varies 60–80%.
• Free promotions end. The :free suffix means a limited window. The paid variant (tencent/hy3) is $0.20/$0.80 per million tokens — cheaper than DeepSeek at nominal rates, but DeepSeek's aggressive cache pricing ($0.003625/M cache-read vs Hy3's $0.05/M) flips the comparison above ~88% cache hit.
• Rate limits constrain throughput. 16 req/min is fine for a single-pipeline dev session; it's a bottleneck for batch evaluation or parallel coding lanes.
• The reviewer comparison uses existing data. The other models were benchmarked days earlier. API behavior can drift. The relative ranking is the signal, not the absolute numbers.

The headline finding — most efficient coder, weakest conceptual reviewer — reinforces what every role-isolation probe has shown: role→model assignment can't be intuited from a model's general capability. You have to measure each role in isolation. A model that writes flawless code might not be able to tell you why someone else's code is wrong.