How to reduce LLM API costs without buying more failures

If you’re working out how to reduce LLM API costs, start with accepted work, not token prices. A cheaper agent that needs more retries can cost more to operate. This is the same discipline as making workload costs visible: measure the outcome you actually need.

Use the Successful Task Ledger, a three-part framework for AI agent cost optimisation:

1. Account for every attempt, including unsuccessful and abandoned work.
2. Replay comparable tasks so changes face the same workload.
3. Admit only changes that pass quality gates, defined before the experiment.

API cost per successful task = total billed API spend across the cohort ÷ accepted completed tasks. Include failed runs, retries, fallbacks and abandoned work in the numerator. Define acceptance against task requirements, not whether the agent returned an answer. With zero successes, report the result as undefined, never zero cost.

Arithmetic example: compare two runs of the same 100-task cohort.

The lower bill buys worse unit economics. Keep the wider business measure separate: (API spend + tool infrastructure + human remediation) ÷ accepted tasks. Otherwise, savings in Claude API costs can disappear into engineer time fixing incomplete work.

Cloudflare’s 19 October 2026 case study reported review costs falling from $2.43 to $0.77, a reported 68% reduction. That is evidence from its code-review workload, not a savings forecast for yours. Its performance reviewer also lost important findings at medium effort. That trade-off makes quality-adjusted accounting essential: lower LLM cost per task means little if the task no longer passes.

Wrap your existing agent; do not replace it. This synchronous ledger separates agent token usage from task outcomes, making LLM cost per task measurable.

python -m venv .venv
. .venv/bin/activate
python -m pip install anthropic
python -m pip freeze > requirements.lock

Set ANTHROPIC_API_KEY in the process environment; the client reads it automatically. Keep real keys out of source control. Disable SDK retries: Anthropic otherwise retries selected errors automatically, hiding attempts from your agent’s retry controller.

import json
import time
from anthropic import Anthropic

client = Anthropic(max_retries=0, timeout=60.0)

def emit(row):
    with open('ledger.jsonl', 'a', encoding='utf-8') as f:
        f.write(json.dumps(row) + '\n')

def measured_call(task_id, variant, call_id, attempt, **request):
    start = time.perf_counter()
    row = dict(event='call', task_id=task_id, variant=variant,
               call_id=call_id, attempt=attempt,
               is_retry=attempt > 0, model=request['model'])
    try:
        response = client.messages.create(**request)
        usage = response.usage.model_dump()
        writes = usage.get('cache_creation') or {}
        row.update(
            uncached_input=usage['input_tokens'],
            cache_write=usage.get('cache_creation_input_tokens', 0),
            cache_write_5m=writes.get('ephemeral_5m_input_tokens'),
            cache_write_1h=writes.get('ephemeral_1h_input_tokens'),
            cache_read=usage.get('cache_read_input_tokens', 0),
            output=usage['output_tokens'],
            request_id=getattr(response, '_request_id', None),
            stop_reason=response.stop_reason,
            usage_known=True,
        )
        return response
    except Exception as exc:
        row.update(error=type(exc).__name__, usage_known=False)
        raise
    finally:
        row['latency_ms'] = round((time.perf_counter() - start) * 1000)
        emit(row)

def finish_task(task_id, variant, start, success, tool_calls):
    emit(dict(event='task', task_id=task_id, variant=variant,
              success=bool(success), tool_calls=tool_calls,
              latency_ms=round((time.perf_counter() - start) * 1000)))

Route every model request through measured_call, including fallback calls. Start attempt at zero; increment it only when retrying the same logical call_id. A different model-routing step gets its own call ID.

task_id, variant = 'extract-001', 'baseline'
start = time.perf_counter()
success = False
tool_calls = 0
try:
    response = measured_call(
        task_id, variant, call_id='extract', attempt=0,
        model='claude-sonnet-5-5',
        max_tokens=4096,
        messages=[{
            'role': 'user',
            'content': 'Extract invoice number as JSON only: '
                       '{"invoice_number": "..."}. Text: Invoice INV-104.'
        }],
    )
    text = ''.join(
        block.text for block in response.content if block.type == 'text'
    )
    success = json.loads(text) == {'invoice_number': 'INV-104'}
finally:
    finish_task(task_id, variant, start, success, tool_calls)

The outer runner calls finish_task exactly once, after its acceptance checker, including on failure. Start its timer before tools and retries: end-to-end latency includes tools and backoff. Success means meeting the task rubric, not receiving HTTP 200.

• Token accounting: Claude’s input_tokens excludes cache reads and writes. Do not subtract them again.
• Prompt caching costs: price writes by TTL, checking the breakdown sums to cache_write. Missing breakdowns remain unresolved, not silently five-minute writes. Use the model-specific Claude API rates.
• Failures: missing usage means unknown cost, not zero. Report that count and reconcile it against provider billing.
• Experiment metadata: record prompt version, effort, provider and rate-card date against each variant. Log metadata, never customer prompts.

For optional language background, see our Python and Node.js comparison.

Fix the experiment price date at 21 October 2026. Use standard, first-party Claude API prices in USD per million tokens. Exclude batch discounts, negotiated discounts, tool fees and regional premiums.

This dated rate card uses the provider’s pricing page. Keep Haiku’s prompt-size tiers separate when comparing Claude API costs.

Calculate each call using five disjoint token buckets:

C=(U×Pi+W5×Pw5+W60×Pw60+R×Pr+O×Po)/1,000,000

U is uncached input; W5 and W60 are cache writes; R is reads; O is output. Each P is the corresponding rate. Sum all call costs, including failed attempts and retries, before dividing by successful tasks.

Propose a pilot of 100 sanitised tasks, stratified by complexity and risk, with three repeats per variant. Keep a separate held-out acceptance set. This is a practical pilot size, not proof of statistical power.

Freeze tool responses or sandbox state, and keep the task mix constant. Change one thing at a time:

• A: preserve production behaviour.
• B: change only prompt ordering and cache boundaries.
• C: change only unnecessary tool round trips.
• D: change only effort.
• E: change only model.

Randomise execution order and isolate experiment prefixes so variants cannot warm each other’s caches. Measure deliberate cold starts and reuse at realistic arrival intervals. Include warm-up spend; a permanently warm benchmark understates prompt caching costs.

Reproducible results template — not an experiment already run. Record one row per variant; task latency includes retries.

Combine passing changes afterwards, then rerun the combination against the held-out acceptance set.

Structure requests as: stable tool definitions → stable system instructions and reference material → explicit cache boundary → changing customer input and tool results. Put cache_control={"type":"ephemeral"} on the final stable content block. For the one-hour experiment, use {"type":"ephemeral","ttl":"1h"} as its value.

A timestamp before the boundary can invalidate reuse. Keep request-specific metadata after it. Cache lifetime refreshes on reuse.

Sonnet 5.5 and Haiku 5.5 have a 512-token cacheable-prefix minimum. Check actual cache_creation_input_tokens and cache_read_input_tokens; the marker does not guarantee a hit. The prompt caching documentation explains configuration and usage accounting.

Derive break-even from prompt caching costs. For a fixed prefix of T tokens, let Pw, Pr and Pi denote per-million-token write, read and uncached input prices. One write followed by h reads costs:

Cached:   T×(Pw+h×Pr)/1,000,000
Uncached: T×(1+h)×Pi/1,000,000
Caching wins when: h>(Pw−Pi)/(Pi−Pr)

Using section three’s dated Sonnet 5.5 rates, five-minute caching needs h>0.263: one subsequent read wins. One-hour caching needs h>1.053: two subsequent reads win.

These calculated examples cover only the prefix, excluding the unchanged suffix and output. The one-hour option loses with only one read.

Skip caching when reusable material is insufficient, documents are unique, prefixes change, or arrivals fall outside the reuse window. Do not pad every prompt to reach a threshold. That buys tokens, not better LLM cost per task.

KV/prompt caching reuses computation; response caching reuses an answer and needs separate freshness and authorisation rules. DigitalOcean’s prefix-aware GPU routing concerns self-hosted serving. Its cache locality does not establish managed API billing savings.

AI agent cost optimisation starts with less repeated work, not shorter prompts. For a support investigation, compare five sequential log lookups and five model resumptions against one bounded lookup returning the same required evidence. Use the same task and acceptance rubric; record LLM round trips separately from tool calls.

• Deduplicate identical lookups within the investigation’s fixed time window.
• Filter large JSON responses to required fields. Preserve request IDs, Kubernetes pod identifiers, timestamps and escalation evidence.
• Batch independent, read-only queries where supported. Don’t merge dependent calls or change their semantics.
• Use Python for deterministic parsing and formatting, rather than another model turn.

Parallel tool calls do not automatically reduce token charges. Price any new retrieval, filtering or summarisation calls too: moving work elsewhere does not make it free.

Hypothetical Sonnet calculation: assuming $0.10 per million cached tokens read, 10,000 cached tokens on each of six resumptions cost $0.006 in prefix reads; two resumptions cost $0.002. This illustrative rate is not a quoted Sonnet tariff; use the applicable model-specific Claude API rates. It excludes writes, dynamic context and output, so it is not a full-task savings claim.

A smaller tool result is not success if filtering removed the error that should have triggered escalation. Judge LLM cost per task against accepted outcomes, not response size.

Set explicit ceilings: eight steps, six tool calls, two retries and 60 seconds elapsed. Protect side-effecting tools with idempotency keys. Log transient transport retries separately from retries after failed answer validation, with distinct reasons. These controls belong alongside the operational agent workflows used in cloud operations.

Separate model routing from effort selection. Routing chooses the model; effort controls how that model works. Compare medium against high on the same Sonnet workload, changing nothing else:

output_config={"effort":"medium"}

Effort influences thinking, tool calls and other output—not just visible-answer length. Keep max_tokens as a safety bound. Evaluate truncation; do not lower it until the bill looks better.

Synthetic arithmetic, not measurements: 100-task cohorts using the experiment’s dated provider price card. Token volumes below are millions: uncached input/five-minute cache writes/cache reads/output (U/W5/R/O); W60=0 throughout.

These volumes explain the calculator; they are not expected agent token usage or claimed quality results. Every Haiku request must remain within its stated tier.

Choose gates before testing:

• At least 95% accepted tasks.
• No newly introduced critical failures in the held-out critical set.
• The existing p95 latency SLO.

These are example business gates, not universal benchmarks. Acceptance combines schema checks, task-specific correctness, verified evidence and blinded human review where needed. Inspect confidence intervals and high-risk slices before calling a small difference real.

Route bounded extraction cheaply only when those checks pass. Send ambiguous or high-risk tasks to the stronger configuration. Count router calls and failed cheap-first attempts in total LLM cost per task. Reject a downgrade that fails the quality gate, even when it wins the arithmetic.

A winning experiment needs a monthly budget, not just a cheaper benchmark. Estimate savings as forecast task volume × measured reduction in cost per successful task, minus new tool, evaluation and infrastructure costs.

For example, 100,000 tasks saving £0.012 each gives £1,200 gross savings; £300 of additional monthly costs leaves £900. This is a planning estimate, not a promise. Recalculate for different task mixes and cache reuse rates: fewer repeated prefixes can erase the saving.

On AWS, use this path: application agent → measured provider adapter → JSON ledger → Amazon S3, queried with Athena. Emit aggregate metrics to CloudWatch or Prometheus/Grafana. Keep task IDs in logs, never high-cardinality metric labels.

• Track spend per accepted task, failure rate and retry rate.
• Track cache writes, cache reads and p95 latency.
• Break these down by workflow and configuration.

For a Bedrock Converse adapter, map inputTokens to uncached input. Put cacheReadInputTokens and cacheWriteInputTokens in separate buckets; retain cacheDetails for TTLs. Use Bedrock’s own region/model rate card, not first-party Claude API prices.

Canary the change on 5% of traffic for 24 hours, with an immediately reversible configuration. Roll back on quality-gate or latency-budget breaches. Prefer reversible prompt and orchestration fixes before adding model routing infrastructure the workload cannot justify.

Prioritisation rule: among variants passing quality gates, rank estimated monthly net savings against engineering effort and operational risk.

For a clearer diagnosis and prioritised options, talk to a cloud engineer about an AI workload cost-and-reliability review covering unit economics, agent behaviour and deployment controls. No promise of Cloudflare-sized savings.

Start with one workflow, a representative task set and a ledger that includes failed attempts. Restructure stable prompts, remove unnecessary tool rounds, then test model and effort changes against explicit quality gates. Keep only the changes that lower cost per successful task without breaching your reliability requirements. If the bill still lacks a clear explanation, Cloudline Consulting can help you discuss an AI workload cost-and-reliability review and identify the next investigation worth doing.