How to reduce AI running costs without sacrificing quality
Reuse context, improve instructions and test effort levels against the work your business actually needs.
An AI assistant can spend money doing work that adds little to its answer. It might process the same reference material repeatedly, search for information it already has, or spend too long considering a straightforward question. Those are useful places to investigate when an application becomes expensive to run.
The aim is to reduce the cost of completing a task to the standard your business needs. Three changes can help. Reuse repeated context through prompt caching, give the assistant clear and current instructions, and test how much reasoning effort different tasks require. Confirm that the assistant still meets the same quality standard after each change.
Start with one familiar task
Imagine a customer asking a support assistant whether they can return a pair of shoes. The assistant needs the relevant policy, enough information about the order, and a clear understanding of what it can decide itself. It may need to ask a question or refer an exception to a person.
A good answer explains the next step accurately. A cheap answer that invents a return deadline has failed. An expensive answer that searches the same policy repeatedly has also created avoidable work. Looking at the whole interaction helps you distinguish those cases.
Record the current cost, response time and outcome before making changes. Include unsuccessful attempts and any retries. Otherwise, a setup can appear cheaper simply because its failures have disappeared from the calculation.
Reuse the context that stays the same
Prompt caching lets a provider reuse processing already completed for the beginning of a request. That beginning is called a prefix. For our support assistant, it might contain shared instructions and the returns policy. The customer's question follows afterwards, and the model generates a fresh answer.
Claude requires an exact match for the cached prefix. Keep reusable material first and changing values, such as a request timestamp, later. If a timestamp changes before the policy, that policy can lose its chance to benefit from the existing cache.
Caches also expire. Claude offers five-minute and one-hour options, with different write charges. Creating a cache costs more than ordinary input processing, while reading an eligible existing cache costs less. Prompt-length requirements and prices vary by model. Reuse frequency determines whether the arrangement saves money.
In our example, the team should use the current approved policy even when an update requires fresh processing. Keeping outdated information to preserve a saving would undermine the purpose of the assistant. A policy version recorded with each answer also makes it easier to investigate a disputed response.
Make instructions clear enough to act on
Instructions often grow as teams respond to individual mistakes. A rule is added after a poor answer, then another after an exception. Eventually, a simple returns enquiry may inherit directions intended for several unrelated situations. Review the complete instruction set when changing models or workflows.
Consider an instruction asking the assistant to search thoroughly, check everything twice and always give a comprehensive explanation. It leaves the assistant guessing which evidence matters and how much explanation the customer needs. It also says little about when the task is complete.
The right wording depends on the business. A useful review asks whether each instruction describes an actual requirement, whether two rules disagree, and whether examples still demonstrate the behaviour you want. Keep examples that clarify a difficult case. Remove or rewrite examples that encourage unnecessary work.
Preserve the checks that protect the result
A refund decision still needs the relevant policy and order facts. An action that requires approval still needs that approval. Reducing cost should preserve those requirements and the evidence used to satisfy them.
Suppose the assistant has just retrieved the correct order, confirmed that the result is current, and has all the fields needed for the decision. An immediate identical lookup may add nothing. If the order changed, the first lookup failed, or the result is incomplete, another lookup may be necessary.
Ask what uncertainty each check resolves. A check with a clear purpose deserves a place in the workflow. A repeat action triggered only by vague wording deserves investigation. Test the change on cases where a second check really is needed, so the assistant retains that ability.
Match reasoning effort to the task
Some models expose an effort setting that influences how much work they do before responding. Lower effort can reduce token use and response time, but it can also reduce capability. It is a behaviour setting, rather than a guaranteed spending limit. Available levels and recommended defaults vary by model.
A routine question about a clearly stated return window is a candidate for testing at lower effort. A complaint involving multiple orders, conflicting evidence and a policy exception may need more investigation. These are ideas to test. Measure the outcomes before assigning settings to live enquiries.
Read cost-versus-quality charts by checking the task and model first. In a published test on FrontierCode Diamond, a benchmark containing 50 particularly difficult coding tasks, Claude Fable 5 scored 11.5% at low effort for $5.35 per task. At maximum effort it scored 30.9% for $19.00 per task. That is a 19.4 percentage-point improvement for about 3.6 times the cost. Extra effort helped on this test, but neither setting solved every task.
Other tasks may gain much less from extra reasoning. Small differences can also reflect variation between runs. A benchmark can guide an experiment, but it cannot establish the best setting for your support inbox. Measure the cost and quality of your own work before changing the settings.
Use a scorecard that can catch a bad saving
Build a set of realistic enquiries covering ordinary requests, missing information and exceptions. Decide what a successful response must contain before comparing settings. Keep a separate set of examples out of the tuning process, then use it to check whether the changes work on new examples.
- Answer quality. Does the reply use the right policy and explain the correct next step?
- Required actions. Were necessary checks completed and required approvals followed?
- Cost per successful task. Divide the total measured usage cost, including failed attempts and retries, by the number of tasks that meet the standard.
- Response time. Check typical interactions and the slower cases that leave customers waiting.
- Failures. Look for missed exceptions, unnecessary escalations, invented facts and incomplete answers.
Keep the same test cases and scoring rules across each comparison. Change one main factor at a time where practical, so you can explain the result. Repeat close comparisons and inspect individual failures. A healthy overall average can conceal a serious problem with one type of enquiry.
Make the decision against the business standard
Our support team might find that caching reduces repeated input costs while answers behave as before. It might discover that clearer instructions prevent duplicate searches. It might also find that lower effort works for routine questions but misses important details in complaints. Each finding supports a different decision.
Choose changes that reduce measured cost while meeting the agreed quality requirements. Keep a record of the model, instructions, settings and test results, then watch real outcomes after rollout. Recheck when the model, policy or mix of customer enquiries changes. The useful saving is the one your business can sustain while customers continue to receive the right help.