The 38.8% Ceiling: What Real-SWE's Private-Codebase Benchmark Means for How You Evaluate AI Coding Agents

Vendors love to lead with a number. “Our model resolves 81% of SWE-bench Pro tasks.” “State-of-the-art on SWE-bench Verified.” Those numbers are real, and they keep climbing. But this week, a new benchmark called Real-SWE from Specific Labs asked a much less flattering question: what happens when you run the same frontier coding agents against the kind of codebase your engineers actually work in — private, undocumented, full of company-specific conventions nobody wrote down? The best model-and-harness combination in the entire test resolved 38.8% of tasks. Everything else scored lower, down to 16.2%.

That gap — from claims in the 80% range on public leaderboards to under 40% on real production code — is the actual story here, and it has direct implications for how you evaluate, budget for, and pilot any AI coding agent your team is considering. This post walks through what Real-SWE tested, why the results looked the way they did, and what to change about your own evaluation process before you sign a contract based on a vendor’s benchmark slide.

What Real-SWE Actually Tested

Specific Labs built Real-SWE by licensing production codebases from real companies — including a Luma/Partiful-style events app with 200,000+ users and a top-100 App Store ranking, a consumer fintech platform that processes over 100,000 bank statements, and enterprise sales platforms running complex, multi-service business logic. According to Specific Labs’ own benchmark writeup, the tasks weren’t synthetic puzzles — they were real engineering work with real business consequences: getting sales-tax calculations right so exempt customers aren’t taxed, migrating customer records without breaking downstream systems, fixing billing logic that spans several services.

The methodology matters here. Each task ran eight independent times per model, evaluated as pass@1 with 95% confidence intervals, inside isolated sandboxes using the Harbor task format, with verifiers built from the codebase’s own existing test suites. The median task required editing 11 files — nearly double the 6-file median on comparable benchmarks like FrontierCode and DeepSWE — and came with a median instruction length of 1,742 characters, reflecting how much unstated context a real ticket actually carries.

Why Private Code Changes Everything

The single most important design choice in Real-SWE is that none of this code exists on the public internet. As Specific Labs put it, these tasks “are not available anywhere on the internet” and were very unlikely to have been part of any model’s training data. That’s a deliberate contrast with public benchmarks built on open-source GitHub repositories, which — after two years of every major lab training on public code — carry a real risk of contamination: models that have effectively memorized parts of the solution space rather than reasoning through it fresh. Real-SWE forces every model to actually navigate someone else’s undocumented system for the first time, the same way a new hire would on day one.

The Numbers: Public Leaderboard vs. Private Codebase

The published rankings tell the story on their own. Fable 5.1, running inside Claude Code, topped Real-SWE at 38.8%. GPT-6 Astra followed at 33.8%, Gemini 3.8 Flash at 31.2%, GLM 5.3 at 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 at 18.8%, and GPT-5.6 Sol trailed the field at 16.2%.

Now compare that to the same models’ public benchmark scores. On SWE-bench Pro’s public leaderboard, Fable 5.1 leads at 81.2% — more than double its Real-SWE result. That’s not a fluke of one model; it’s the entire point of the exercise. A benchmark score built on public repositories is measuring something real, but it is not measuring “how this model will perform on the codebase you’re about to point it at.”

Where the Failures Actually Come From

Real-SWE classified failures using a taxonomy adapted from DeepSWE’s failure categories: unverified assumptions, missed requirements, integration errors, regressions, and edits to the wrong file entirely. Across most models, missed requirements was the dominant failure mode — the agent implemented something plausible-looking that simply didn’t satisfy what the ticket actually asked for. Failure rates were high across the board: 71.4% of rollouts completed in under ten minutes still failed, and 73.4% of longer-running attempts failed too, which suggests the problem isn’t primarily about giving the model more time — it’s about the model not correctly understanding the system it’s operating in.

This lines up with a pattern we’ve already flagged in GitHub’s HydraFusion release, where quality on repository-level, cross-file tasks was consistently the weakest spot even for a sophisticated multi-model router. Real-SWE’s median of 11 edited files per task is exactly the kind of cross-file, context-heavy work that keeps exposing this gap.

Cost Doesn’t Buy You Accuracy

One more result deserves attention: estimated per-rollout cost ranged from $2.50 for Gemini 3.8 Flash to $6.96 for Fable 5.1, and cost did not track cleanly with resolution rate. The most expensive model to run was also the best performer here, but the relationship wasn’t linear or guaranteed across the rest of the field — some cheaper configurations outperformed pricier ones. That’s consistent with what we found covering the AI coding agent price war: “cost per task” is the number that actually matters for budgeting, and it has to be measured on tasks that resemble your own work, not a vendor’s demo suite.

The Procurement Lesson Nobody’s RFP Covers

None of this means AI coding agents aren’t worth deploying — plenty of Real-SWE’s example failures still produced code that was mergeable with meaningful human review, and the field is improving version over version. But it does mean the number on a vendor’s benchmark slide answers a narrower question than most buyers assume. It tells you how a model performs on tasks that resemble its training distribution. It does not tell you how it will perform on your authentication service, your billing pipeline, or the fifteen-year-old module nobody wants to touch.

We’ve made a version of this point before: Microsoft’s ThinkingBox work showed that one successful run tells you almost nothing about repeatable reliability, and the GPT-5.6 benchmark-gaming incident showed that benchmark scores themselves can be actively gamed. Real-SWE adds a third, more structural problem: even an honestly-run, ungamed benchmark score built on public code is measuring the wrong distribution if your codebase isn’t public.

What to Actually Do Before You Buy

A few concrete changes to how you evaluate coding agents follow directly from this:

• Don’t accept a public benchmark score as your acceptance criteria. Ask vendors directly whether they’ve been evaluated on anything resembling a private, production codebase, and treat a “no” as a real gap, not a formality.
• Pilot on your own repository before signing. A short, structured pilot — a handful of real tickets from your own backlog, run through the candidate tool, reviewed by your own engineers — will tell you more than any leaderboard. Real-SWE’s ten-task, eight-run-per-task design is a reasonable template to scale down for an internal pilot.
• Weight “missed requirements” failures heavily in review. If the dominant failure mode across nearly every frontier model is producing plausible code that doesn’t do what was actually asked, your review process needs to check requirement coverage explicitly, not just “does it run.”
• Track cost per resolved task, not cost per call. The cheapest model per token can easily be the most expensive model per shipped fix once you account for how often it fails — a distinction that matters just as much for the build-vs-buy decision many enterprises are weighing right now.

What This Means Going Forward

Real-SWE is a young benchmark — eight model-harness combinations and ten tasks is a modest sample, and critics on its own Hacker News discussion thread have pointed out that the private nature of the codebases makes full reproducibility impossible for outside researchers to verify independently. That’s a fair caveat, and it’s also somewhat unavoidable: the entire value of the benchmark comes from using code nobody could have trained on, which necessarily means nobody outside Specific Labs can fully audit the underlying tasks either. Treat the specific percentages as directionally useful rather than as a permanent scoreboard.

But the core finding doesn’t depend on the exact numbers holding steady release over release: performance on your own, unseen, convention-laden codebase is a materially different — and currently much harder — problem than performance on public benchmarks, for every frontier model tested. That gap isn’t going to close because a lab ships a new model next quarter. It closes when your evaluation process stops taking a vendor’s number at face value and starts measuring the thing you actually care about: how the tool performs on the code you’re going to ask it to touch.

Frequently Asked Questions

What is the Real-SWE benchmark?

Real-SWE is a benchmark from Specific Labs, published in September 2026, that evaluates frontier AI coding models on private, production codebases licensed from real companies, rather than the public GitHub repositories most coding benchmarks use.

Why did the best model only resolve 38.8% of tasks when it scores over 80% on other benchmarks?

Because Real-SWE deliberately uses codebases that never appeared in any model’s training data and require understanding company-specific conventions, undocumented dependencies, and cross-file changes — conditions that public benchmarks built on well-known open-source repositories don’t fully capture.

Which model performed best on Real-SWE?

Fable 5.1, running inside the Claude Code harness, led the field at a 38.8% resolution rate, ahead of GPT-6 Astra (33.8%), Gemini 3.8 Flash (31.2%), and five other model-harness combinations scoring between 16.2% and 28.8%.

Does a higher-cost model perform better on Real-SWE?

Not reliably. Estimated per-task cost ranged from $2.50 to $6.96 across the tested models, and while the top performer was also the most expensive to run, cost and resolution rate did not track cleanly across the rest of the field.

What should enterprises change about how they evaluate AI coding agents?

Run a short pilot using real tickets from your own codebase before committing to a tool, rather than relying solely on a vendor’s public benchmark score, and pay particular attention to “missed requirement” failures, which were the most common failure mode across nearly every model tested.

Is Real-SWE’s methodology fully verifiable by outside researchers?

Not entirely — because the benchmark’s value depends on using codebases that were never publicly available, external researchers can’t fully audit the underlying tasks the way they can with open benchmarks, a limitation Specific Labs and outside commenters have both acknowledged.

Sources

• Real-SWE Benchmark - Specific Labs - Official benchmark methodology, task design, and full model-and-harness leaderboard.
• Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases - Hacker News - Community discussion, model-specific observations, and methodology critiques.
• SWE-bench Pro Leaderboard: Claude Fable 5.1 Takes #1 at 81.2% (2026) - CodingFleet - Public benchmark comparison data used to contrast against Real-SWE’s private-codebase results.
• Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases - The Daily Commit - Additional coverage and summary of the benchmark’s findings.
• Specific Labs - Y Combinator - Background on Specific Labs, the company behind the Real-SWE benchmark.