Cutting LLM Costs Without Cutting Quality: 9 Levers That Move the Bill
First, find out where the money actually goes
Almost nobody can answer 'which feature costs the most' on day one. Before optimising anything, tag every model call with a feature name, a user or tenant ID and a request type, then log input and output tokens per call. Nine times out of ten the bill is concentrated: one background summarisation job or one over-eager retrieval step is responsible for the majority of spend, and it is not the feature anyone was worried about. A day of instrumentation saves weeks of guessing.
Prompt caching is the cheapest win available
If you send the same long system prompt, tool definitions or document context repeatedly, caching that prefix cuts the cost of those tokens dramatically and reduces latency at the same time. The engineering work is mostly restructuring prompts so the stable part comes first and the variable part comes last. For chat products and document Q&A this alone routinely takes 30–50% off. It is one of the few optimisations with essentially no quality trade-off, which is why it should be step one after instrumentation.
Stop sending context nobody reads
Retrieval systems drift toward sending more chunks 'just in case'. Measure it: pull the top ten chunks, then check how often the answer actually cites anything below rank four. Usually it doesn't. Cutting from ten chunks to four, with a reranker in front, tends to improve answer quality and cut input tokens by half at once. Same principle applies to conversation history — summarise older turns rather than replaying the whole transcript on every message.
Route by difficulty
Not every request needs your most capable model. A classifier — often a small model, sometimes just heuristics on request length and type — can send routine requests to a cheaper tier and escalate only the hard ones. Clients running this pattern typically send 60–80% of traffic to the cheap tier with no measurable drop in user-visible quality. The important part is measuring: define your eval set first, then tune the routing threshold against it rather than by feel.
Batch anything that isn't interactive
Overnight enrichment, backfills, document classification, report generation — none of that needs a synchronous response. Batch processing APIs are substantially cheaper per token, and the only cost is accepting results within a window rather than in seconds. Most enterprises have more offline AI workload than they realise; it is worth auditing which jobs a user is genuinely waiting on.
Cap the output, not just the input
Output tokens usually cost several times more than input tokens, and models are happy to pad. Setting sensible max token limits, asking for structured output instead of prose, and dropping 'explain your reasoning' from prompts where nobody reads the reasoning all reduce spend directly. If you need reasoning for debugging, log it in a sampled 5% of requests rather than all of them.
The remaining levers, briefly
Cache full responses for identical or near-identical requests — FAQ-style traffic has more repetition than you expect. Deduplicate concurrent identical calls. Fail fast with retry budgets so a flaky endpoint does not triple your call volume. And renegotiate: at meaningful volume, committed-use pricing is available from every major provider, and most teams never ask. Together these levers have taken client bills down 40–70% without a single user complaint about quality.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch