Don’t Kill The Golden Goose: Rectifying AI Cost Without Stifling Innovation

This voice experience is generated by AI. Learn more.This voice experience is generated by AI. Learn more.Mark Williamson, CTO at Undo, builds agentic debugging to fix complex non-deterministic bugs — likes to remind computers who’s in charge.

getty​For two years, the AI story in software engineering has been simple: adopt fast, spend freely and trust the returns will follow. Engineering leaders were rewarded for getting developers onto AI tools, token spend became a badge of progress and the question of what it was producing was set aside.

That story is running out of road. According to TechCrunch, Uber exhausted its entire 2026 AI budget by April, primarily on Claude Code. Axios reported that an AI consultant’s unnamed client spent roughly $500 million on Claude in a month after failing to set usage limits.

CFOs are asking what the business got for that spend. Over the next 12 to 24 months, that scrutiny is going to turn into budget pressure for most engineering organizations. But there’s a danger in how companies respond.

On the way up, the mistake was treating token consumption as progress. Companies built dashboards tracking prompts, tokens and active users. Some ran leaderboards ranking teams by usage.

​Google’s own usage figures illustrate how dramatically AI token consumption has grown: The company reported processing more than 3.2 quadrillion tokens per month in May 2026, compared with 9.7 trillion two years earlier. But usage volume alone doesn’t establish business or engineering value.​

For many companies, all those AI tokens don’t necessarily translate into better productivity. In fact, Gartner found that over half of developers say AI has improved team output by 10% or less. A team can double its AI usage without shipping any faster.​

Now the emerging mistake is treating token cost as the thing to control. Cap the spend, mandate cheaper models, push budgets onto individual teams and let them ration.​

These look like opposite strategies, but they’re really two sides of the same coin. Both treat the token as the unit that matters. Neither asks what a token actually buys. The better measure is the cost of solving the problem. And once you price things that way, some of the “obvious” cost savings turn out to be nothing of the sort.​

Many companies now have an internal mandate that amounts to using the cheaper model wherever possible, and saving the frontier model for special occasions. On paper this is sensible. The workflows look identical and one model costs a fraction of the other per token.​

Engineers, meanwhile, are grumbling that the cheaper model isn’t up to the harder problems. It’s tempting for leadership to hear that as engineers wanting the shiniest toy. But model choice can miss the bigger variable. What makes a model effective on a given task often has less to do with the model than the context it’s given.​

On a complex debugging task, for example, even the most expensive AI models can struggle to identify the root cause of a bug if they only have the source code. Context on the runtime behavior of the software is also essential.​

Taking issue #20086 in DuckDB (a widely used open-source database) as an example, testing across different models and contexts, my company discovered:

• Claude Sonnet, source code only spent $4.35 and gave a confident wrong diagnosis.

• Claude Fable 5, source code only spent over an hour and $37.60 and still hallucinated the root cause.

• Claude Fable 5 with runtime context found the real root cause in 12 minutes, for $2.88 to $6.31 across runs.

• Claude Opus 4.8 with runtime context found it in 14 minutes, for $8.06.​

First, this wasn’t a cheap-model problem. With source code alone, both Sonnet and Fable 5 confidently got the root cause wrong. The real variable was context rather than model.

Second, with runtime context, the model gap nearly disappeared. The cheaper model performed close to the more capable one and beat that same model running on source alone.

Third, better context changed the economics completely. Fable 5 went from $37.60 and wrong to $2.88 and correct—a six- to 13-times cost reduction.

That’s why token price is the wrong metric. A cheap failed run still leaves the bug unsolved. What’s important is cost per solved problem or, put differently, token ROI.

There are two levers engineering leaders can pull to optimize the ROI of AI, and neither is as simple as buying cheaper tokens.​

1. The first lever is matching the model to the task. Cheaper models are genuinely the right choice for a lot of engineering work, such as boilerplate code and mechanical refactors. Frontier models are better suited for harder reasoning tasks where a wrong answer is expensive, like debugging unexpected system behavior. A blanket “use the cheap one” mandate eliminates the judgment call that makes spend efficient.

2. The second lever is engineering the context so the model succeeds first time. An AI agent with the right context behaves like a smarter model. Feed it a full recording of everything the code did at runtime, rather than asking it to guess dynamic behavior from logs and static source code, and it stops hallucinating plausible-but-wrong causes and starts finding real ones.​

Even with both levers pulled, engineering leaders need to know whether AI spend is working, and the answer will never be on a token dashboard. The questions worth asking are the boring ones. Has MTTR for customer issues come down? Are features shipping faster? Have escaped defects dropped? If those numbers haven’t moved, the spend isn’t working, regardless of how much AI is nominally in the process. If they have, defend the budget with them.​

The budget crunch is coming, and there are two ways to respond. Organizations that ration tokens could pay twice: once for cheap tokens, and again for the wrong answers and the engineer-hours spent chasing them. Those that respond by pricing the solved problem, matching models to tasks and giving agents the context to succeed first time will get the productivity gains everyone signed up for, at a bill their CFO can live with.​

The golden goose doesn’t need a bigger feed budget, but to stop being judged by how much it eats.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?