Guide · Costs
11 ways to cut your LLM bill without making the product worse.
LLM bills grow quietly: longer prompts, more retrieved context, more calls per request, a bigger model “just to be safe”. Most of that spend buys nothing. Here are eleven levers, roughly in the order we pull them — each one checked against an evaluation set so quality doesn’t drop with the cost.
By Nythrex EngineeringUpdated 3 min read
System prompt & instructions
Retrieved context (RAG)
Conversation history
User message
Output
Illustrative breakdown of tokens per request in a typical RAG assistant — not measured data. Measure your own before optimising.
The 11 levers
- 1
1. Measure per request, per component
Log tokens for system prompt, context, history and output, per call. You can’t optimise an aggregate number on an invoice.
- 2
2. Use prompt caching
Keep stable content (instructions, tool definitions, static knowledge) at the start of the prompt so providers can cache it and bill repeated prefixes at a reduced rate.
- 3
3. Route by difficulty
Send simple requests (classification, extraction, short answers) to a small, fast model and escalate only hard ones to a large model. Often the single biggest saving.
- 4
4. Retrieve less, retrieve better
Better ranking lets you send three strong passages instead of ten mediocre ones. A reranker usually pays for itself.
- 5
5. Summarise conversation history
Replace old turns with a running summary instead of resending the whole chat each time.
- 6
6. Trim the system prompt
Prompts accumulate rules nobody remembers adding. Remove, merge and test — every token is paid on every call.
- 7
7. Cap and shape the output
Set sensible maximum output lengths and ask for concise, structured output. Output tokens are usually the most expensive.
- 8
8. Cache whole answers
Identical or near-identical questions (FAQ-style) can be answered from a cache, with invalidation when the source changes.
- 9
9. Batch offline work
Nightly classification or enrichment can often use providers’ batch interfaces at lower prices and without latency pressure.
- 10
10. Fine-tune a small model for a narrow step
At high volume, a small model tuned on reviewed examples can replace a large general one for one specific task. See RAG vs fine-tuning.
- 11
11. Enforce budgets in code
Per-request, per-user and per-day limits with alerts. A runaway agent loop or a scraper hitting your chatbot should trip a limit, not your finance team.
LLM API cost calculator
System prompt + retrieved context + user message
Example price tiers — replace them with your provider’s current price list.
Cached input is typically billed at a fraction of the normal price; we assume 10%.
Estimated monthly cost
$251
/ month
Per request
$0.0042
Per 1,000 requests
$4.19
Same traffic on each example tier
Biggest levers: shorter system prompts, fewer retrieved chunks, caching, and routing easy requests to a smaller model.
Already live and the bill keeps growing? We review prompts, retrieval and model routing and tell you where the money goes.
Get a cost reviewDon’t trade quality for cost by accident
Every lever above can reduce quality if applied blindly: a smaller model misses nuance, less context misses the answer, a shorter prompt drops an important rule. That’s why each change should run against your evaluation set, with cost and quality reported side by side. Read LLM evals for non-ML teams.
Low cost → High cost
Frequently asked questions
Keep reading
Want a second opinion on your project?
Tell us what you’re building and where you’re stuck. We’ll reply within one business day with the most practical next step — even if that step isn’t us.