🔍 Read the full analysis: FrontierCode: The Questions Behind Reviewing An AI Pull Request on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that judges whether a code change would meet repository maintainers’ standards, not just pass tests. Claude Opus 5.5 scored highest, but cleared all blocking checks in 57.7% of attempts; the results also show large cost differences and gaps between general intelligence rankings and coding-review performance.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a pull request meets repository maintainers’ standards, and found that the top-scoring model cleared every blocking check in 57.7% of attempts. Claude Opus 5.5 led with a 52.2% benchmark score, but the results show that even the strongest model tested did not consistently produce changes that passed all required checks.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026. It ran the models through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The evaluation used the benchmark’s 100-task Main set; the full FrontierCode benchmark contains 150 tasks.
Claude Opus 5.5 (max with fallback) ranked first at 52.2%, followed by GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The benchmark score is distinct from the rate of clearing all blocking checks: Opus did so in 57.7% of attempts, GPT-6.1 Sol in 54.7%, and Fable in 54.3%. Artificial Analysis reported that nine of the 100 tasks were not solved by any model on any attempt.
FrontierCode assesses patches using hidden tests, command checks, tests that must fail on the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria. A failed blocking check makes an attempt score zero. The review stage used GPT-6 Luna (medium) as its judge. These design choices make the score a measure of more than whether code runs: the benchmark also penalizes changes that exceed scope or include weak tests.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
The results put a number on a practical concern for teams adopting coding agents: a green test run does not by itself make a pull request ready to merge. FrontierCode includes checks for scope, regression risk, test quality and alignment with maintainers’ requirements. On this evaluation, the leading model failed to clear at least one blocking check in roughly four out of ten attempts, based on its reported 57.7% pass rate.
Cost also varies sharply among models. Artificial Analysis reported a cost of $10.15 per task for Opus and $0.81 for GPT-6.1 Sol, while the latter scored 3.1 percentage points lower. The figures are benchmark-specific and do not establish what a deployment would cost in a particular company’s workflow. They do show why buyers may weigh reliability and price together rather than selecting a model from its top-line score alone.
The ranking also differs from general intelligence results. For example, Claude Sonnet 5.5 ranked second among the listed models on Artificial Analysis’ Intelligence Index but fifth on FrontierCode; Grok 4.7 fell to 18th on FrontierCode. That contrast suggests general-purpose rankings may not predict performance on repository-level code review tasks.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Patches
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules intended to prevent models from finding solutions online. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task, while the tasks themselves are private so models could not have encountered them during training.
The benchmark’s core distinction is between passing tests and producing a reviewable change. It checks whether an agent’s own tests distinguish the original bug from the patched code, whether the patch stays within file and diff limits, and whether it satisfies maintainers’ criteria. Attempts flagged by Cognition’s trajectory scanner for looking up a solution receive zero, according to the benchmark description.
Artificial Analysis’ results cover only the Main set of 100 tasks, not all 150. The source material also lists models without results as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. The leaderboard is therefore a snapshot of the models tested at that time, not a complete comparison of every coding model.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of This Snapshot
The results do not establish how models would perform across every repository, programming language or company workflow. Artificial Analysis tested 100 of FrontierCode’s 150 tasks, and the source does not provide confidence intervals or enough detail here to determine how sensitive close rankings are to task selection. Scores should be read as outcomes on this particular benchmark rather than guarantees of production performance.
Several models had no result as of October 13, including Gemini 4 Argon and Muse Wonder Full, so comparisons may change when more evaluations are added. The cost figures are reported per task, but the source does not specify a deployment-specific estimate for repeated use, human review or other workflow costs. It also remains unclear how closely the benchmark’s LLM review judge aligns with different maintainers’ decisions outside the tested tasks.
As an affiliate, we earn on qualifying purchases.
More Models and Results Ahead
The immediate next development is the addition of results for models absent from the October 13 leaderboard. Artificial Analysis’ published snapshot identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as lacking FrontierCode results at that point. Their eventual scores could shift comparisons, though no release schedule is specified in the source material.
Readers comparing coding agents should watch for further benchmark runs and check whether they use the same task set, harness, attempt limits and judging process. Until results expand, the current figures support a narrower conclusion: on FrontierCode’s maintainers’ criteria, the leading tested systems still failed blocking checks in a substantial share of attempts, so human review remains relevant to interpreting these benchmark results.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI-generated code change meets repository-level criteria, including hidden tests, regression checks, scope limits and an LLM review against maintainers’ requirements. A patch can lose credit even if it passes ordinary tests.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’ October 2026 results, with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does a 52.2% score mean Opus passed 52.2% of attempts?
No. The reported 52.2% benchmark score and the 57.7% rate for clearing all blocking checks are separate measures. The source describes scores as depending on whether checks pass and on the quality checks applied.
How much did the leading models cost per task?
Artificial Analysis reported $10.15 per task for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol. These are benchmark cost figures, not estimates for every real-world deployment.
Does the benchmark prove AI pull requests are ready to merge without review?
No. The results are limited to FrontierCode’s task set and evaluation setup. The leading model cleared all blocking checks in 57.7% of attempts, so the benchmark does not show consistent success across the attempts it tested.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
