The Hidden Math Behind Your LLM API Bill

The AI feature shipped. It works. Three months later the API bill is four times the original estimate and nobody can explain the difference. This is the pattern we see most often when auditing production LLM integrations: the cost model was never designed — it was inherited from the prototype, and the prototype was never meant to scale.

The fix is usually not switching providers or dropping features. It is a handful of implementation decisions, most of which take a day or less to apply.

What You're Actually Paying For

LLM providers charge by the token — roughly by the word, including punctuation and whitespace. The important detail: you pay for both sides of the conversation. Every token you send (the prompt) and every token the model returns (the completion). In most production workloads, the prompt is far more expensive than the completion.

In a customer-facing chat feature, conversation history is typically included in every request so the model has context. If a user sends ten messages averaging 50 tokens each, by message ten you are sending 500 tokens of history plus your system prompt plus the current message — on every single call. The token count compounds with every turn.

Prompt Bloat Is the First Place to Look

System prompts are the most common culprit. We have audited production integrations with system prompts of 2,000–4,000 tokens — detailed instructions, edge cases, company background — prepended to every API call. A feature handling 10,000 requests per day at 3,000 tokens of system prompt alone is spending 30 million input tokens on overhead before a user has typed a word.

Audit your system prompt. Remove anything that explains behaviour the model already follows by default, adds context that is never referenced, or duplicates instructions. The test is simple: remove one section at a time and check whether outputs change. In most cases they don't.

Then look at conversation history. You almost never need the full history. Most tasks work with the last three to five turns. A sliding window with a hard token cap takes a day to implement and typically reduces context tokens by 60–70% in a long-running chat feature.

Prompt Caching: The Optimization Most Teams Skip

Both major LLM providers offer prompt caching: a mechanism that lets you mark a static portion of the prompt so the provider caches it server-side, charging a fraction of the normal input token rate on cache hits. For features with long, stable system prompts — a support assistant, a document-analysis pipeline, an internal knowledge tool — the numbers are significant. Anthropic charges roughly 10% of the standard input token rate on cache hits; OpenAI is in a similar range.

The implementation is a single parameter change in the SDK. If you have a 2,000-token system prompt that repeats on every call and your cache hit rate is 80%, you have reduced your effective input spend on that portion by 80%. Most teams building LLM features today are not using prompt caching. Most teams that start using it reduce monthly API spend by 30–50% in the first billing cycle.

Model Tiering: Not Every Request Needs Your Best Model

The cost difference between a frontier model and a mid-tier model from the same provider is typically 5–10x per token. The capability difference, for most production tasks, is smaller than that gap suggests.

Classification, intent detection, short-form summarisation, and routing decisions don't need a frontier model. A smaller model handles them at a fraction of the cost with comparable accuracy. If you process 100,000 requests per day and 60% are routing or classification tasks, the spend difference between tiers is material. Build your integration with the model as a configuration parameter, route by task complexity, and test the quality tradeoff empirically before committing. This is not premature optimisation — it is a one-time setup that pays for itself in weeks.

When Your Bill Is a Proxy for a Bug

Cost monitoring reveals a specific class of bugs that functional test suites miss. We inherited one production integration where the average completion length was 3,000 tokens. The feature was a short-answer Q&A tool. The root cause: no max_tokens constraint on the completion. The model was generating full explanations by default because nothing told it not to. One parameter change reduced completion token spend on that feature by 80%.

Instrument token usage per feature and per request before anything else. A spike in average prompt length usually means a code change is injecting context it shouldn't. A spike in completion length usually means a prompt change broke the expected output format. Your cost data is a proxy for prompt health — and it catches problems that look invisible in functional testing.

These are not exotic optimisations. Every team we have worked with that built LLM features into a production product has benefited from at least two of them. The order of operations matters: instrument first, identify the biggest source of waste, fix it. The savings compound because each change reduces the baseline for the next.

Rebooted Solutions audits LLM integrations as part of our AI Audit service — reviewing token budgets, caching strategies, model selection, and prompt architecture in production systems. If your API bill is growing faster than your user count, get in touch to scope a session.