🔍 Read the full analysis: FrontierCode: Where Maintainers Draw The Line On AI Contributions on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis published results for 23 models on FrontierCode, a coding benchmark that grades patches against maintainer-style review criteria, not just test results. Claude Opus 5.5 led, but cleared all blocking checks in 57.7% of attempts; the results also show large differences in cost and performance among models.
Artificial Analysis published results for 23 AI models on FrontierCode, a coding benchmark designed to measure whether maintainers would accept a proposed change, rather than whether it merely passes tests. Claude Opus 5.5 ranked first at 52.2%, but it cleared every blocking check in 57.7% of attempts, leaving substantial room for review and correction.
Artificial Analysis added the evaluation on October 12, 2026, using its 100-task Main set. Claude Opus 5.5 (max with fallback) scored 52.2%, followed by GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The reported score is distinct from the pass rate: Opus cleared all blocking checks in 57.7% of attempts, while Sol did so in 54.7% and Fable in 54.3%.
FrontierCode evaluates changes across six areas, including hidden tests, regression and cleanliness checks, tests that must fail on the original code, scope limits, and an AI code review against maintainer criteria. Any failed blocking check gives an attempt a score of zero; passing all blocking checks produces a score from 70% to 100%, depending on quality checks. Attempts flagged by Cognition’s trajectory scanner for looking up solutions also receive zero.
Artificial Analysis ran all models through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the reviewer. The benchmark’s full set contains 150 tasks; the Main set used for these results contains 100. Nine of those 100 tasks were not solved by any model on any attempt, according to the source report.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results measure a gap between producing code that works under tests and producing a change that meets a repository’s review standards. FrontierCode’s checks penalize issues such as oversized diffs, edits outside relevant files, and tests that do not demonstrate the intended behavior. These are concerns maintainers face when deciding whether to merge a contribution, not just whether software runs.
The leading model’s 57.7% blocking-check pass rate is a benchmark result, not evidence that AI-generated pull requests can be accepted without human review. The source report argues that the figure does not support claims of reliable autonomous pull requests. More cautiously, the data show that, under this evaluation’s setup, even the top-ranked model failed at least one blocking check in a substantial share of attempts.
Performance also comes with a marked cost difference. Artificial Analysis put Opus at $10.15 per task and GPT-6.1 Sol at $0.81. The report says Sol-to-Opus buys 3.1 percentage points for about 12.5 times the cost. Those figures give teams a benchmark-specific comparison, but they do not establish what a model will cost or achieve in a different repository, workflow, or task mix.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Changes
FrontierCode was created by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against searching online for task solutions. According to the source report, more than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy, and Mattermost, spending more than 40 hours on each task. The tasks are private, intended to reduce the chance that models encountered them during training.
The benchmark’s central distinction is its acceptance-oriented grading. Passing tests alone is insufficient: checks examine correctness, regressions, the quality of tests, the patch’s scope, and alignment with maintainers’ review criteria. Tomer Nosrati, a Celery maintainer, described that approach in Cognition’s announcement: “Where others grade like a CI, FrontierCode grades like a tech lead.”
The rankings also differ from general model leaderboards. In the comparison reported by Artificial Analysis, Claude Sonnet 5.5 ranked second among the listed models on its Intelligence Index but fifth on FrontierCode. Grok 4.7 ranked 18th on FrontierCode in the broader results, while DeepSeek V4.1 Flash and GPT-6 Luna placed higher there than their general intelligence scores might suggest. That comparison indicates that general scores and repository-patch performance are not interchangeable.
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
software development code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The results do not establish how models will perform across all repositories or real-world pull requests. Artificial Analysis tested the 100-task Main set under a shared harness and fixed attempt limits; the published figures should be read as outcomes under those conditions. The source report does not provide enough detail here to assess how closely benchmark task frequency or difficulty matches any particular development team’s workload.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every model that might be relevant to coding work. It is also unclear from the supplied material when those models may be tested or whether the benchmark’s task set and scoring will change.
The report gives per-task cost estimates, but the figures are tied to the benchmark runs and do not by themselves describe total deployment expenses, human review time, or the cost of fixing failed patches. No conclusion about whether a model is cost-effective for a specific team follows from the leaderboard alone.
As an affiliate, we earn on qualifying purchases.
Missing Models and Further Runs
The next notable development will be results for the models absent from the October 13 comparison, particularly Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. The source report identifies Muse Wonder Full as a model to watch because it performed well on Terminal-Bench and the Coding Agents Index, but it provides no FrontierCode result for that model yet.
Readers should also watch for additional benchmark runs and any changes to the task set or scoring rules. Until such results are available, the current leaderboard offers a comparison of 23 models under one stated evaluation setup, not a final measure of how much human review AI coding tools require in everyday software development.
open-source code analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-generated repository changes meet a set of maintainer-oriented checks, including hidden tests, regression checks, patch scope, and code review criteria. A change that passes ordinary tests can still fail a blocking check.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a 52.2% FrontierCode score and a 57.7% rate of clearing all blocking checks.
Does the top score mean AI pull requests are ready to merge without review?
No. The benchmark reports that the leading model cleared every blocking check in 57.7% of attempts under its test setup. That result does not show that AI-generated changes can be accepted without human review in real projects.
How much did the leading models cost per task?
The source report lists Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81. These are benchmark-specific figures, not estimates of the full cost of using either model in a development team’s workflow.
Were all major coding models included?
No. As of October 13, 2026, the source report listed no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full, or Mistral Large 4 Preview.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
