CursorBench 3.1 turns coding-assistant hype into a scoreboard
1 arrives with a fairly plain-spoken promise: stop grading coding assistants on tidy little toy problems and see how they do when the work looks more like a real software job. That means tasks rooted in actual growth work, the sort that usually live in messy repositories rather than polished demo clips.
Because of this, it’s a timely reset. The coding-assistant market has spent the last year throwing around big claims about faster shipping, along with smarter agents and fewer late-night debugging sessions. Some of that may be true in practice. Some of it may be marketing with a headset on. 1 tries to pull a few of those claims down to earth by putting them on a scoreboard.
The first number people will notice is a little awkward, in the way honest numbers usually are. The strongest model clears the low-70s on accuracy. That’s a decent result, but it’s not the kind of score that lets anyone declare victory and head home early. Just as awkward, the best-performing systems do not all come with anything close to the same price tag. One can sit near the top while charging many times less per task than another model with a similar grade.
A coding benchmark means less if it only proves a model can handle a polished puzzle; the real test is whether it can survive contact with actual software.
That distinction matters for the people deciding whether to buy, build, or ban these tools. Developers want something that helps without creating a second job in cleanup. Teams paying for seats and usage want numbers that tell them whether a model is a useful helper or just an expensive confidence machine. And anyone watching the assistant wars unfold gets a clearer view of which products are actually earning their keep.
There’s also a broader tech news angle here, even if nobody in a meeting room would phrase it that way. Benchmarks like this give buyers a way to compare claims that otherwise blur together. One company says its assistant writes better code. Another says its model plans better, reviews better, or handles long tasks with fewer stumbles. Big difference, and fine. Show the score. Show the cost. Show the tradeoff. That’s the part people can argue about without turning the room into a demo theater.
1 is trying to do exactly that: replace vibes with measurements. Not perfectly, of course. No benchmark captures every real-world annoyance, and a score never tells the whole story about a tool’s usefulness. But when the conversation around coding assistants gets this loud, even a rough yardstick beats a stack of promises.
Then the useful question has changed too. It isn’t just whether a model can code at all. It’s whether it can do it without burning through money fast enough to make procurement people reach for the aspirin. 1 update changes what gets tested, and that changes how seriously the numbers should be read.
What changed in the 3.1 update
1 doesn’t just hand out a fresh batch of scores. It changes the shape of the test itself. That matters, because a benchmark is only as useful as the work it actually asks models to do. If the task list’s too neat, AI coding assistants can look clever in a way that doesn’t survive contact with a real repo, where the files are messy, the conventions are uneven, and the bug report is usually written by someone who’s already mildly annoyed.
0 version leaned hard on edit, refactor, and bug-fix problems. Those are useful, sure, but they mostly test whether a model can patch up code that has already been framed for it. 1 widens the lens. It adds work around understanding codebases the model hasn’t seen before, tracing bugs through unfamiliar files, planning multi-step changes, and reviewing code rather than simply rewriting it.
The real test for AI coding assistants is no longer whether they can edit code. It’s whether they can walk into a strange repository, figure out what matters, and avoid making a mess.
That shift sounds small until you think about what it changes in practice. A model that can rename a function or fix a typo may still flounder when it has to understand why a bug exists, where the logic lives, and which change would cause the fewest future headaches. 1 tries to measure that broader judgment.
The new task mix also changes the flavor of the scorecard. Bug fixing is still in the mix, but now it sits beside repository understanding and change planning, which require a different kind of attention. A model can’t coast on surface-level pattern matching quite as easily if it has to infer project structure, spot the right files to inspect, and think ahead about side effects. That makes the benchmark feel less like a synthetic exam and more like a stand-in for the sort of problem a team might toss at a junior engineer who has just joined the project and is trying not to ask the same question twice.
The grading got stricter, too. 1, so the benchmark is Broader but less forgiving about sloppy output. That kind of change matters because it closes a loophole that benchmarks often leave open. If the scoring rubric is too loose, a model can appear competent while producing edits that are technically close but functionally awkward, or fixes that look fine until a reviewer opens the diff and sighs into their coffee. Tighter grading forces the benchmark to care more about whether the answer actually works in context, not whether it merely resembles a plausible answer.
For readers following AI policy, digital culture, or just the day-to-day churn of lifestyle tech, this is the boring part that turns out not to be boring at all: the benchmark itself’s getting more realistic. The task set is broader, the judgment is sharper, and the whole thing is less friendly to flashy demos. Cursor’s own CursorBench page frames the benchmark around practical software work, which is the right direction if the goal is to measure something people might actually pay for. A polished benchmark can still be gamed, of course. No scoring system gets to escape that fate. 1 makes the game harder to play with smoke and mirrors.
That matters because the question has shifted. Teams are no longer just asking whether AI coding assistants can produce code that compiles. They want to know whether the model can step into a repo, read the room, and keep its edits from turning into a support ticket later. 1 is trying to measure that, one task at a time. And yes, that means the assistant that breezes through a refactor may still get humbled by a weird old bug buried three folders deep, which is honestly a very human outcome.
The leaderboard: strong models, steep bills
1 leaderboard does something a lot of coding-assistant marketing would rather avoid. In the best possible way, it puts accuracy and cost on the same page, which is rude. Anyone who has spent time with SWE-bench will recognize the basic shape of the problem: a model can sound sharp in a demo, then fumble when it has to work through real repository tasks instead of chatting about them (and that’s no small thing).
At the top of this software engineering benchmark sits Fable 5 Max, just under three-quarters accurate. That sounds strong, and it is. It also lands at the most expensive end of the table, at roughly eighteen dollars per task. For a team evaluating developer tools, that number changes the conversation fast. A model that gets more answers right can still be a hard sell if every run looks like a small line item from finance.
But the next Fable 5 settings stay near the front of the pack. Maybe, their scores sit in the low to mid 70s, but the price eases down from the mid-teens into the high single digits as you move through the variants. That spread matters because it shows how much a buyer can pay for a few extra points. In ordinary software work, a handful of percentage points can be the difference between a useful AI code review assistant and a very expensive typo machine. On paper, the top Fable runs look tidy. In practice, the bill arrives with a little attitude.
The sharpest model on the board is not automatically the one your team can afford to run every day.
A second cluster sits lower on the accuracy chart but starts to look friendlier on spend. 5 settings land in the mid-60s. One of them comes in at well under five dollars per task, which is a very different proposition from the top Fable run. That gap is the whole story here. Once you line the models up this way, the choice stops being about who can get closest to perfect and starts being about how much imperfection you can live with for a given budget.
5, which might be the most awkwardly practical entry on the board. It reaches the low 60s while costing only around half a dollar per task (which is worth thinking about). That’s a wild spread. It means a team could run a lot more experiments, do more internal testing, or simply afford to keep the thing on in production without feeling like every prompt is nibbling at the monthly budget. Cursor’s own Composer 2 technical report gives some background on the model family, but the benchmark result is probably the part that matters here. It suggests that the cheapest useful option may not be a toy at all.
The bottom half of the table is less flattering. Lower-end Sonnet and Kimi settings slide into the 40s and low 30s. That’s the point where the benchmark stops being polite. It’s one thing for a model to miss an obscure edge case or two. It’s another for it to fall apart once the tasks involve unfamiliar code, bug tracking, planning, and code review. At those levels, the assistant may still be fine for simple scaffolding or quick rewrites, but it no longer looks like a peer that can keep up in a messy codebase.
The cost numbers here deserve a quick translation. They’re not random sticker prices pulled out of a hat. CursorBench calculates them from each model’s published token pricing, including input, cache, and output usage, then averages the result across tasks. That matters because a model that looks cheap at first glance can behave very differently once you account for how often it pulls cached context, how much it emits, and how much prompt material it needs to chew through to stay competent. Simple as that, and pricing tables love to be tidy. Real workloads usually aren’t.
Naturally, for teams trying to buy or approve AI coding assistants, that makes the leaderboard more useful than a single headline score. A model in the mid-70s may be the right answer for a team doing heavy repo work or shipping code under tight review. 5, might be the better choice for broader rollout, internal tooling, or lighter AI code review workflows. And the models down in the 40s and 30s? They may still have narrow use cases, but they are no longer in the same conversation as the front-runners.
The punchline is simple enough. 1 shows that coding assistants aren’t marching in a neat line toward the same destination. CursorBench 3.1 shows that coding assistants aren’t marching in a neat line toward the same destination. Some are cheaper. No surprise there. A few are both decent and affordable. The fun part, if you enjoy spreadsheets with a little existential anxiety, is that the best answer changes depending on whether you care more about raw performance or about keeping the cloud bill from developing a personality.
What the scores really tell teams shipping code
By the time a benchmark gets this specific, the temptation is to treat the leaderboard like a verdict. Fable 5 Max at the top, a few other strong settings close behind, cheaper models farther down, case closed, right? (believe it or not). Not quite. 1 seem aware of that trap, which is why the small gaps at the top shouldn’t be read like gospel. A couple of points here or there can wobble with task mix, prompt shape, or plain old variance. If one model edges another by a hair, that doesn’t automatically mean it’ll feel better in a real engineering workflow on Tuesday afternoon.
A narrow score gap can look like a clean win on paper and vanish once the code gets messy.
That caveat matters because the useful comparison here isn’t just raw accuracy. It’s capability against cost, and that part is harder to hand-wave away. Some models clearly do better on practical coding tasks, but model pricing changes the story fast. A team that spends a lot of time on bug finding, review, and codebase understanding might decide the pricier option earns its keep. Another team, especially one with repetitive internal tooling or narrower tasks, might get almost all the value it needs from a cheaper configuration that lands a few points lower. If the assistant saves time on the boring parts and doesn’t produce a steady stream of cleanup work, that can be enough.
After that, the new task mix also makes the benchmark more believable for day-to-day software work. Earlier versions leaned harder on tidy edit and refactor jobs. 1 asks models to make sense of unfamiliar repositories, trace bugs, plan changes, and read code like a teammate instead of a tutorial robot. That matters for teams deciding whether to buy seats for individual developers or roll an assistant out across a whole org. A model that can patch a function in a clean sandbox is one thing. A model that can reason through a real codebase with odd naming, half-finished abstractions, and the occasional mystery import is something else entirely.
That’s where the benchmark becomes more than a brag sheet. It gives buyers a way to talk about tradeoffs without drifting into pure impressionism. The cheapest useful model may be the smart move, if a product team spends its mornings in one service. If an systems group is juggling multiple repos and needs better code review support, paying more for stronger performance could make sense. The point is not that the most expensive option wins by default. When it comes to the point, it is that the bill and the output now sit in the same conversation, which is healthier than the old routine of trusting demo clips and polished adjectives.
The broader market will probably feel that pressure. Coding assistants are getting compared on work that looks closer to actual engineering, and that makes vague claims harder to sell. Fair enough. Teams can still disagree on what “good enough” means, of course. They always will. But they’ll have a cleaner set of numbers in front of them, and those numbers can be tied back to the jobs people are trying to get done.
For buyers, that’s the real shift here. The race is no longer about who can sound most magical on launch day. It’s about which assistant can handle the repo, the bug report, and the budget without causing a scene.
