LLM Cost Optimization: Reduce AI Costs

The most effective way to cut LLM costs is not hunting for a cheaper model, but seeing what every request is spent on and removing tokens the system does not need. In practice most savings come from five places: caching repeated context, routing easy work to smaller models, trimming the context sent to the model, running non-urgent work in batches, and, once volume justifies it, running open-weight models on your own infrastructure. None of these can be managed properly unless cost is measured per successful task rather than per call.

This guide is for product owners, technical leads and CTOs who already run an AI feature in production and are wondering what to do as the bill grows. We first look at where the cost actually comes from, then at when each optimization technique pays off and when it starts to hurt quality. All figures in this article are illustrative; provider prices change often, so run your own calculation against the current price list.

Where exactly does LLM cost come from?

The invoice may show a single "tokens" line, but AI cost is the sum of several different items. Optimizing without knowing which one is largest usually means investing in the wrong place.

Input tokens: The system prompt, tool definitions, document chunks retrieved through RAG, conversation history and the user's question. In most production systems this is the largest item, because the same long context is sent again with every request.
Output tokens: The text the model generates. Their unit price is usually higher than input. Needlessly long answers, formats that ask for explanations and reasoning tokens all grow this item.
Retries and failures: Second and third calls caused by invalid JSON, timeouts or answers that fail validation. Invisible, but they add up.
Agent loops: A tool-using agent can make ten or more model calls for a single user request. Context grows at every step, so cost rises faster than the number of steps.
Embeddings and retrieval: Turning documents into vectors, the vector database and reranking models.
Hosting and operations: If you run models yourself, the GPUs, scaling, monitoring and on-call time.
Human review: Team time spent correcting low-confidence answers. The real cost of a model that looks cheap on paper is often hidden here.

Why measure cost per successful task instead of per call?

Cost per call is misleading, because a cheap call that fails to do its job is not free. The right metric is the total cost of an outcome that actually solves the user's task, including failed attempts, retries and human correction.

Illustrative calculation (made-up figures): In a document classification task, model A costs 1 unit per call and model B costs 4 units. Model A gets 70 percent of tasks right; the remaining 30 percent are fixed by an expert whose time costs 10 units. Model B gets 97 percent right. The average cost per task is 1 + 0.30 × 10 = 4 units for A and 4 + 0.03 × 10 = 4.3 units for B. The model that is four times cheaper on paper ends up costing almost the same per task, while leaving your team with far more corrections. As the cost of an error grows, the balance tips towards the more expensive model.

Doing this calculation requires two things: a test set that measures accuracy, and a monitoring layer that records which task each request belongs to and whether it succeeded. Without them you cannot measure the effect of any of the techniques below either.

How does caching reduce LLM cost?

Caching simply means not paying for the same work twice. There are three kinds, and each carries a different risk.

Prompt caching (prefix caching)

Most large model providers cache context that starts identically across consecutive requests and process that part at a lower price or faster. If the system prompt, tool definitions and fixed reference documents are the same in every request, this is the easiest saving available, with no loss in quality. The rule is simple: content that does not change goes at the start of the context, content that changes (the user's question, the date, session data) goes at the end. Putting a timestamp accurate to the second at the very top of the system prompt is a common mistake that breaks the cache on every request. Providers differ in cache lifetime, minimum length and pricing, so check your provider's documentation and track the hit rate in your logs.

Exact response cache

For work where the same input should produce the same answer (generating a product description, translating fixed text, summarizing a document once), storing the answer itself means not calling the model at all. The key should combine a hash of the input, the model name, the prompt version and the parameters, so that old answers are invalidated automatically when the prompt changes.

Semantic cache

A semantic cache matches questions that are not identical but similar in meaning, using embedding similarity, and returns the earlier answer. It can save a lot where the same questions are asked in different words, such as a support bot, but it is the riskiest kind of cache. "Can I cancel my order?" and "I can't cancel my order" look similar but need different answers. Answers containing personal information may be served to another user, which is a serious privacy problem under GDPR and similar laws. Use a semantic cache only for general information answers, with a strict similarity threshold, for entries that contain no user-specific data, and with time-based invalidation. Set the threshold with a test set of real question pairs, not by guesswork.

How do model routing and cascading work?

Sending every request to the most powerful model is like shipping every parcel by air. A large share of incoming work is classification, field extraction, short summaries or format conversion, which smaller models handle comfortably. There are two common patterns:

Routing: The request is sent to the right model up front, based on its type. This can be done with a simple rule set (task type, input length, customer tier) or with a small classifier model. Starting with rules is cheap and easy to debug.
Cascading: The request goes to the small model first. If the answer passes a check (schema validation, a business rule, a confidence score, a second quick check), it is used; if not, it is escalated to the large model. Most easy tasks finish in the cheap tier, and hard tasks still get the right answer.

The price of routing is measurement. Which work can be left to a small model must be decided with a test set, not by intuition. When a provider ships a new version, the same test set is run again and the routing rules are updated. A model-agnostic calling layer (one internal interface with swappable providers behind it) is the precondition for this flexibility.

What do token budgets and context trimming gain?

You do not pay for tokens you do not send. The typical reasons context swells in production systems, and their fixes:

Too many RAG chunks: Sending 20 document chunks "just in case" raises both cost and the risk of relying on the wrong passage. Picking the few most relevant chunks with reranking is usually both cheaper and more accurate.
Unlimited conversation history: In long chats, summarize older messages or keep only the last few turns plus important persistent facts.
Unused tool definitions: Instead of sending an agent the definitions of 40 tools with every request, give it the subset relevant to the task type.
Uncontrolled output: Set a maximum output length, ask for structured output (a JSON schema) and use instructions like "explain step by step", which lengthen answers, only where they are truly needed.
Agent step limits: Give every agent task a step and token budget. When the budget is exceeded, the agent stops and hands over to a human; an agent stuck in a loop can burn a monthly budget overnight.

The effect of trimming on quality must always be measured with the test set. Overly aggressive trimming produces cheap but wrong answers, and the cost per successful task metric shows this immediately.

When does batch processing make sense?

Any work where the user is not waiting for the answer is a candidate for batch processing: overnight document classification, enriching a product catalog, labeling historical records, regenerating embeddings, evaluation runs. Most large providers offer a discounted interface for batch requests that accept delivery within a few hours. The point to watch is that the workflow can genuinely wait: if a user is waiting on screen, batch is not suitable. Batch jobs also need failed items to be re-queued and results to be validated.

When are open-weight models on your own servers cheaper?

Open-weight models have become much stronger over the past year. With Qwen 3.8, a Max-class model shipped with open weights for the first time, and models such as Kimi K3 have come close to closed models on some tasks. That turns "API or our own servers?" into a real cost question.

The two cost structures are fundamentally different. With an API, cost is variable: you pay for what you use, and nothing when you use nothing. On your own servers, cost is largely fixed: the GPU server, the engineering time that keeps it running, monitoring and redundancy are paid whether there is traffic or not. That is why self-hosting only becomes cheaper at high and steady volume.

Illustrative calculation (made-up figures): Compare a setup with a fixed monthly cost of 100 units (GPU, operations and monitoring included) with an API that charges 2 units per million tokens. The break-even point is 50 million tokens per month. Below it the API is cheaper; above it, as long as the server has capacity, your own setup wins. In a real calculation, include the server's average utilization too: the cost of a GPU sitting idle at night lands on the daytime requests.

Beyond cost, two reasons can favor your own servers: cases where personal data must not leave for a foreign provider because of data protection rules or data residency, and latency or availability requirements. The structure we recommend most often is hybrid: sensitive or high-volume, relatively simple work runs on an open model on your servers, while rare and difficult work goes to a strong model through an API. The routing layer unifies both behind a single interface.

How should you monitor LLM cost?

Cost you do not measure cannot be managed. For every model call, at least the following should be recorded: which feature or workflow it served, which customer or tenant it belongs to, the model and prompt version used, the input, output and cached token counts, latency, the number of retries and whether the result succeeded. Three views come out of these records:

Cost per feature: How much each product feature spends and what its cost per successful task is.
Cost per customer: Especially in SaaS products, this shows whether your pricing covers the AI cost. A single heavy customer can distort the average.
Anomaly and budget alerts: An alert when daily spend crosses a threshold, quotas per agent or per user. A bug or a loop should be caught before the invoice arrives.

Rather than hard-coding provider prices, keep them in a versioned price table, so that past reports stay correct when prices change and new calculations use the right price. If the monitoring data contains personal data, decide masking and retention up front.

LLM cost optimization checklist

Do you measure cost per successful task rather than per call?
Is every call logged with feature, customer, model and prompt version?
Do you have a test set that measures accuracy and is re-run on every change?
Is the fixed context (system prompt, tool definitions) at the start, so it benefits from prompt caching?
Is an exact response cache used for repeated inputs?
If you use a semantic cache, are answers containing personal data excluded from it?
Is easy work routed to a small model or solved with cascading?
Are the number of RAG chunks and the conversation history limited?
Are a maximum output length and structured output defined?
Do agents have a step and token budget?
Does work that can wait run through a batch interface?
Have you compared API and open-weight models for your volume at current prices?
Are daily budget and anomaly alerts in place?

How does Detartech approach AI cost?

We talk about cost at the start of a project, not at the end. When designing an AI feature, we first put in writing how success will be measured and what cost per task is acceptable. Then we build a model-agnostic calling layer, per-call cost and quality logging, and a test set, and make caching, routing and trimming decisions based on those measurements. Where sensitive data is involved, we also evaluate hybrid setups that run open-weight models on your own infrastructure. For the questions to ask about cost control when choosing a partner, see our guide on how to choose an AI integration company.

If your AI bill is growing faster than your product, or you want to get the cost of a new feature right from the start, see how we work on our AI solutions page or fill in the quick quote form. In the first meeting we map out together where your current usage is being spent.

Frequently Asked Questions

What is LLM cost optimization? It is the work of reducing model usage cost in an AI-powered system without lowering its quality. It covers techniques such as caching, model routing, context trimming, batch processing and, where appropriate, moving to open-weight models, and it is always done together with accuracy measurement.

Does switching to a cheaper model reduce cost? Not always. If the cheaper model makes mistakes more often, retries and human correction can raise the cost per successful task. The decision should be made by comparing both models' accuracy and total cost per task on the same test set.

Does prompt caching affect quality? No. Prompt caching stops the model from reprocessing the same context; it does not change the content of the answer. To be effective, unchanging content must sit at the start of the context and changing content at the end.

When does it make sense to run models on our own servers? When usage volume is high and steady, when data must not leave the country, or when latency and availability requirements are strict. At low and fluctuating volume an API is usually cheaper, because the cost of your own server is fixed even when you do not use it.

How do we track AI cost per customer? By logging every model call together with the customer or tenant ID. These records show how much cost each customer generates and whether your pricing covers it; quotas and alerts are built from the same data.

LLM Cost Optimization: Cutting AI Costs Without Hurting Quality