Most organisations adopting generative AI can answer "what does it do?"
They struggle to answer "what will it cost us?"
That gap is becoming expensive. AI spend is billed per token, and token consumption is far less predictable than most teams expect. A single chatbot conversation might use 200 tokens. A document-heavy RAG workflow can consume over a million. Multiply that by growing usage, retries, and context accumulation across a conversation, and invoice shock becomes almost inevitable.
The good news: token costs are highly controllable once you understand what actually drives them.
Every AI request bills for input tokens (your prompt, instructions, and context) and output tokens (the model's response). Output tokens are consistently more expensive, typically 4 to 5 times the cost of input tokens, and in some premium models, up to 8 times. This means tasks generating long responses, like content creation or code generation, cost disproportionately more than simple classification or extraction tasks.
Multi-turn conversations compound this further. By the tenth turn in a conversation, the cost per call can be roughly 7 times higher than the first turn, simply because context keeps accumulating and gets resent with every message.
Raw API pricing is only the starting point. Realistic budgets need to account for usage growth, infrastructure overhead, experimentation, and peak demand. The practical guidance is to apply a 1.7x to 2x multiplier on top of your base calculated API cost to arrive at a realistic monthly budget.
Skip this step, and most organisations will underestimate spend significantly.
The encouraging part is that meaningful cost reduction doesn't require sacrificing quality. Several proven techniques consistently deliver strong results:
Combined, caching and batch processing can stack to deliver strong compounding savings, though in practice this is often lower than the highest headline figures suggest, largely due to how session behaviour affects caching (see below).
A practical note on caching:
Prompt caching is often enabled automatically in coding tools like GitHub Copilot, but it's not something you can simply set and forget. Switching models, or switching a model's thinking mode, resets the cache. These switches are frequently necessary, different tasks genuinely call for different models or modes, but each switch has a hidden cost: the cached context is lost and effectively paid for again. If you're routing between models within a session, it's worth being deliberate about when you switch versus when you start a fresh session altogether. The cache reset cost is easy to overlook and easy to underestimate, and it's one of the main reasons real-world savings often land below headline figures.
Cost control shouldn't be reactive. The most effective approach follows a repeatable process: instrument every AI request, capture token and cost data, attribute it to the right team or use case, establish a baseline, monitor for spikes, investigate root causes when costs jump, optimise the specific issue, and validate that the fix didn't degrade quality or latency.
This is the same discipline applied to cloud cost management, adapted for a token-based world.
AI token spend is not inherently unpredictable. It's under-instrumented. The figures above represent achievable ceilings under well-instrumented conditions. Real-world savings depend on discipline in how caching, routing, and session management are actually implemented, which is exactly why instrumentation matters more than the headline numbers themselves. Organisations that treat cost visibility as a first-class requirement, not an afterthought, are consistently finding 80–90% in achievable savings, without compromising the experience their AI systems deliver.
If you're heading into your next planning cycle without a clear token budget model, that's the gap worth closing first. Explore our full cost modelling framework and calculators to help close that gap.