Guide
How to measure AI ROI in software engineering
AI ROI in software engineering is the relationship between what you spend on AI coding tools (tokens, seats, gateways) and what you ship, measured in normalized outcomes such as merged PRs or Mentlio Delivery Points and adjusted for quality debt. Enterprise AI ROI surveys from PwC and Deloitte rarely answer the engineering question. Engineering leaders need cost per delivered outcome, not only EBIT narratives. Attribute the spend, pick an outcome unit, compute cost per outcome, then remove waste without capping useful usage.
By Ashank ShahCo-founder, Mentlio (YC S2026)
Why generic AI ROI fails engineering leaders
Most of what gets written about AI ROI is not about engineering. PwC’s 29th Global CEO Survey found that 56% of CEOs had seen no significant financial benefit from AI so far, and only 12% reported both revenue and cost gains. Deloitte’s report on the AI ROI paradox found that 85% of organizations increased their AI investment in the past year while only 6% saw payback within twelve months, with a typical use case taking two to four years to reach a satisfactory return.
Those are useful numbers for a board. They are useless for the question a VP of Engineering actually gets asked, which is some version of “we tripled spend on AI coding tools, so what shipped?” EBIT is the wrong altitude. The AI coding bill arrives monthly, per tool. The work ships daily, per team. Generic frameworks have nothing that connects the two.
What an engineering leader needs is a unit economic. Cost per delivered outcome, measured per team and per workflow, and adjusted for the quality debt that AI-assisted code can carry.
What is AI ROI for software engineering?
AI ROI in software engineering is the ratio of normalized shipped outcomes to fully attributed AI spend over a period, discounted for rework and quality debt. The numerator is what shipped and survived. The denominator is every dollar spent to produce it, including tokens, seats, gateways, and the review and rework hours that AI output creates.
Several metrics get at this. They differ in how hard they are to instrument and how easy they are to game.
| Metric | What it measures | Where it breaks |
|---|---|---|
| AI spend per engineer per month | Tokens, seats, and gateway fees attributed to a person or team | Says nothing about what shipped, and invites tokenmaxxing |
| Cost per merged PR | Attributed AI spend divided by merged pull requests. Unblocked’s write-up adds review and rework cost to the numerator and counts only PRs that survive a churn window | PR size and conventions vary between teams, and the count can be gamed by splitting work |
| Cost per Delivery Point | Attributed AI spend divided by Delivery Points, a normalized unit of shipped work derived from repository evidence | Needs a delivery-scoring model and repository-level evidence |
| Quality-adjusted outcomes | Outcome units discounted for review coverage, revert rate, and incident linkage | Needs review and incident data joined to the same period |
| AI-attributed share of changes | Share of merged changes attributed to AI tools | Attribution is probabilistic. Use it for the trend, not as ground truth |
How to measure AI ROI in six steps
- Inventory the spend. List every AI coding tool in use (Claude Code, Codex, Cursor, GitHub Copilot, internal gateways) and every way it is billed. Seats, API tokens, gateway markups, overages.
- Attribute spend to teams and workflows. The vendor invoice gives you a total. Attribution needs metering where the work happens, on the developer’s device, per session, per tool, per model tier. Roll it up to team and workflow. Do not stop at the individual.
- Choose outcome units and a period. Merged pull requests, production deliveries, or Delivery Points, weekly or monthly. Pick one primary unit and keep it for at least a quarter so the trend stays comparable.
- Compute cost per outcome per team and baseline it. Divide attributed spend by outcome units for each team and period. Run it for four to eight weeks before you change anything, so you know what normal looks like, including how much it moves on its own.
- Adjust for quality debt. Discount outcomes for review coverage, revert and rollback rate, and incident linkage in the same window. A merged PR that is reverted two weeks later was spend with a zero in the denominator.
- Optimize waste, then measure again. Route routine work to cheaper models that are still good enough, cut redundant context, compress logs. Do not cap useful usage. Flag unusual spend for review instead. Then repeat steps 4 and 5 and compare against the baseline.
SURGE vs Delivery Points: how Mentlio’s unit fits AI ROI frameworks
Frameworks like Hivel’s SURGE (Spend, Utilization, Recovery, Gains, Efficiency), co-authored with the DORA lead at Google Cloud, are shaping how people talk about AI ROI in software engineering. They connect AI dollars to shipped code using engineering telemetry rather than survey estimates, and the SURGE framework sits alongside DORA and SPACE instead of replacing them. We agree with that direction. Delivery Points is our named outcome unit inside the same problem. When raw PR counts mislead, normalize the delivered work, compute cost per Delivery Point, and then remove the waste, all without collecting prompts or source code.
What SURGE gets right
- It treats AI ROI as an engineering-systems problem that belongs next to DORA and SPACE, not as a board-level EBIT survey.
- It puts Spend in the denominator, broken down by tool, model, team, and token usage.
- Its Recovery pillar names the waste. Wrong models on routine work, zombie agents, overlapping licenses, over-committed contracts. We go after much of the same waste in the product.
- Its Efficiency pillar uses unit economics that finance already understands, such as cost per PR, AI-assisted PRs merged, and rework rate.
Where Delivery Points sits
| SURGE-style measurement | Mentlio Delivery Points | |
|---|---|---|
| Primary question | What did AI spend return, from Spend through Efficiency? | What did we ship per AI dollar, in comparable units? |
| Common unit | Cost per PR, AI-assisted PRs merged, Efficiency Multiplier | Cost per Delivery Point, where each PR is scored on delivered behavior, technical difficulty, system reach, and validation rather than raw size |
| Waste | Recovery pillar, covering idle seats, tool overlap, wrong-model spend, zombie agents, and over-committed contracts | Token Savers (Route, Lens, Quiet, Logs, Recall) plus spend attributed to each saver |
| Privacy | Read-only telemetry from Git, CI, and AI-tool admin consoles. Hivel states source code is not stored and prompt content is not retained | Classification, scoring, and routing run on the developer’s device. Raw prompts, outputs, and source code never leave it |
| Measure vs optimize | A measurement framework, with Hivel’s product automating it | Measurement plus optional on-device optimization of the spend being measured |
Practical guidance
- If finance already asks for cost per PR, start there. The SURGE chain from Spend through Gains to Efficiency is the right shape for that conversation, and every team can compute cost per PR from data it already has.
- Move to Delivery Points when the PR mix makes raw counts unfair. A one-line config change and a multi-service feature should not count as the same unit. Normalizing delivered work is what keeps cost per outcome comparable across teams.
- Pair either unit with a quality lens. SURGE’s rework rate and our quality adjustment (review coverage, reverts, incident linkage) agree on this. Work that gets rewritten within weeks did not really ship.
- Decide early whether prompts and code may leave the machine. If keeping them off third-party servers is a hard constraint, prefer a stack that can measure and optimize without exfiltrating them.
Whichever unit you pick, the trap to avoid is the one these frameworks exist to prevent. Rewarding usage instead of outcomes is tokenmaxxing.
Delivery Points, defined
Delivery Points are our normalized unit of shipped engineering work. Each merged pull request is scored from repository evidence on four dimensions, delivered behavior, technical difficulty, system reach, and validation completeness. Raw line counts, generated code, and commit or review volume do not add points. The result is one scale on which teams with different PR habits can be compared. A team that ships ten small, reviewed PRs and a team that ships two large ones are measured against the same denominator.
Attributing Delivery Points to AI is observational. We report AI activity and delivery observed in the same window. We do not claim that AI caused every merged pull request. That is on purpose. It keeps the metric honest, and it keeps it useful as a trend.
AI adoption vs AI ROI
Most dashboards blur these two together. They answer different questions, and they fail in opposite directions.
| AI adoption | AI ROI | |
|---|---|---|
| Question answered | Are people using the tools? | Is the spend producing shipped work? |
| Typical metrics | Active seats, tokens consumed, prompts per day | Cost per outcome, adjusted for quality, tracked as a trend |
| Failure mode | Tokenmaxxing, where volume gets mistaken for value | Cost-cutting that caps useful usage and slows delivery |
| Time horizon | Leading indicator, visible in days | Lagging indicator, needs four to eight weeks of baseline |
| Primary owner | Enablement and platform teams | CTO or VP Engineering, together with finance |
Cutting cost without killing ROI
Once cost per outcome is measurable, the temptation is to cut the numerator directly with budgets and caps. That lowers spend, and it usually lowers output with it, so ROI stays flat or gets worse. The better lever is waste, the share of spend that does not change the result.
In A more useful way to manage AI coding costs we describe where that waste lives. The context assembled before the model call, the model tier chosen for the call, and the tool output carried into the next turn. Mentlio’s Token Savers (Route, Lens, Quiet, Logs, Recall) each target one of those layers. In our frozen Terminal-Bench 2.1 replay we saw Mentlio Route retaining 98.93% of Fable 5’s solved-task rate at 24.90% lower average per-task model cost. Route solved 82.92% of tasks at $0.933 per task against Fable 5’s 83.82% at $1.242 per task. That is a replay over a fixed task set with stated assumptions, not a production study. Take it as evidence that cost can come down with the quality floor held, and confirm it on your own traffic before you count it.
The operational side is routing policy, context hygiene, and guardrails that flag unusual spend for review without blocking useful work. How Mentlio measures token-saver savings documents what each saver measures and what its numbers do not claim.
The privacy constraint
Every step above runs on derived data. Token counts, model choice, cost metadata, classifications, and version-control evidence are enough to compute cost per outcome, quality adjustments, and trends. Raw prompts, model outputs, source code, terminal output, and file paths are not needed, and collecting them creates a compliance surface that most engineering leaders would rather not own. Mentlio computes its metrics on the developer’s device and sends only derived metrics to the dashboard.
Frequently asked questions
- What is a good AI ROI for software engineering?
- There is no universal benchmark, and vendor-published multiples rarely survive contact with your own data. Baseline cost per outcome per team, then judge yourself on the trend and on the spread between teams. A falling cost per Delivery Point with stable quality metrics is the result to aim for.
- Is cost per merged PR a good metric?
- It is a good place to start, because every team can compute it from data it already has. Normalize it, for example with Delivery Points, when teams’ PR conventions differ, and adjust for quality by counting only PRs that survive a churn window.
- How long does it take to measure AI ROI?
- Four to eight weeks of baseline before the first intervention, then a comparable window after it. Shorter windows get swamped by release cadence and changes in team composition.
- Should AI ROI include developer time saved?
- Self-reported time savings are noisy and tend to run high. Use them as a secondary signal. Shipped outcomes per dollar, adjusted for quality, is the primary measure because it is observable and hard to game.
- Do we need to read prompts or code to measure AI ROI?
- No. Derived metrics are enough for attribution, outcome units, cost per outcome, and quality adjustment. Mentlio never collects raw prompts, outputs, or source code.
- Is Mentlio a SURGE alternative?
- Not exactly. SURGE is an open AI ROI framework, and Hivel’s own FAQ says you can run it on Git, CI, and AI-tool admin data without Hivel. Mentlio is an engineering intelligence and token optimization platform. It uses Delivery Points as its outcome unit and can cut waste through Token Savers while keeping prompts and code on the developer’s device. Use SURGE’s questions. Use Delivery Points when you need normalized outcomes and privacy-preserving optimization.
Mentlio
Get cost per shipped outcome for your team.
Mentlio attributes AI spend by team, tool, and workflow, computes Delivery Points from repository evidence, and keeps the raw work on the device.
Sources: PwC, 29th Global CEO Survey (January 2026); Deloitte, “AI ROI: The paradox of rising investment and elusive returns”; Unblocked, “Cost per merged PR”; Hivel, “The SURGE Framework” and SURGE framework overview. SURGE is a trademark of Hivel. Our descriptions of it are drawn from Hivel’s public pages. Our benchmark figures are documented on the research pages. They are replay results, not customer case studies.