50-90%. Engineering teams that prioritize prompt caching, model routing and right-sizing can cut LLM API bills by roughly half, and in some workloads nearly eliminate token-level spend. For Canadian teams, start by measuring baseline run-rate and token composition, then prioritize prompt caching, model routing and batching while meeting Quebec privacy obligations. Follow the sequence below to measure, prioritise and implement those levers with traceability and Privacy Impact Assessments in place.

Thousands of tokens. That's the scale of wasted input in one practitioner example, where an assistant produced a short reply while sending a large amount of context on every turn.

1. Measure your baseline, precisely

First, instrument everything before you change architecture. Track cost per 1,000 input and output tokens, requests per feature and the origin of input tokens, because the brief shows input-side tokens usually dominate cost. Break down spend by endpoint and model tier so you can see which features and which users drive the bill. Measure request volume per feature, p95 latency and the share of traffic hitting each model tier. A correct baseline will show you whether system prompts, retrieved documents or conversation history are the primary token drivers.

Worked example: imagine a chat feature that generates a short reply. If the system prompt and retrieved documents add a large repeated context per turn, cutting that repeated context even partly will change monthly run-rate dramatically. Quantify cost per 1,000 input tokens and per 1,000 output tokens so later comparisons are apples to apples.

2. Start with the highest-return changes

Second, put in place the low-risk, high-return levers first. Prompt caching, precise cache invalidation and right-sized routing are the most effective early steps. Prompt caching stores the processed representation of a repeated prefix so you aren't charged to re-process identical tokens for each request. Engineers report that when caching is possible, the repeated-context portion of requests can be nearly fully eliminated from per-call costs, with case studies describing up to 90 percent savings on repeated tokens.

Practical guidance: design precise cache keys that include the user-visible variables you want to vary. Then define explicit invalidation policies for time-sensitive content. Test cached results for freshness and staleness because a high-return caching strategy can fail if cached context masks important updates.

For many chat integrations this single change converts a dominant recurring token cost into near-zero marginal spend on that portion of the request.

Worked example: when identical onboarding text or a stable system prompt appears in every call, save the processed prefix in the cache and only bill for new tokens. If a cached prefix covers 80 percent of tokens per call, and those tokens no longer trigger billing, the per-call cost drops accordingly.

3. Route and right-size models

Third, route each request to the cheapest model that satisfies the quality requirement. The relationship between capability and price is non-linear. Practical audits show frontier models can cost an order of magnitude more than competent smaller models for the same task. Classify requests by intent and confidence requirement, then send low-risk, high-volume calls to small or mid-tier models while reserving high-cost models for genuinely difficult tasks.

Right-sizing often means the 80 percent of routine work runs on a small or mid-tier model, which dramatically reduces aggregate spend. Build routing logic that inspects request type, expected output quality and acceptable latency, and make routing decisions at the API gateway or service mesh level so they're auditable and changeable without redeploying model code.

Worked example: a search summarisation request that needs brief, factual output can go to a mid-tier model. Complex legal summarisation or high-stakes customer support routes should escalate. Measure escalation rates and track whether escalations erase savings.

4. Reuse, semantically and explicitly

Fourth, go beyond exact-prefix caching with semantic caching and response reuse. Semantic caching matches similar requests to previously computed outputs or partial outputs. Combine retrieval-augmented generation with a cache of previously generated answers so you avoid a full model call when an acceptable cached candidate exists. For document-centric use cases, store canonical answers for high-frequency queries and fall back to an LLM only when no cached candidate meets your appropriateness threshold.

Worked example: if customers repeatedly ask the same five questions about an account feature, keep canonical replies and serve them directly. Only when the question deviates beyond a similarity threshold do you invoke the model, and even then you can use a mid-tier model to draft a response that's validated against cached facts.

5. Batch non-urgent work and use asynchronous APIs

Fifth, move non-urgent workloads to batch or asynchronous execution where supported. Providers and practitioners report batch endpoints that process large jobs at substantially reduced token rates, with example discounts around 50 percent for bulk asynchronous processing. Identify nightly re-indexing, bulk enrichment, large-scale evaluations and periodic backfills as primary candidates.

Mastheads

Batch processing reduces peak unit cost and smooths GPU utilisation. When a use case allows time-shifting of inference, queue jobs into bulk pipelines and apply lower-cost compute instances or dedicated batch endpoints. That can cut the unit token cost materially without changing user-facing behaviour.

6. Shrink prompts and trim repeated context

Sixth, apply prompt engineering to reduce input size. Shorten system prompts, trim historical context to the relevant window and use condensed instruction formats. When history is necessary, use summarisation or short-form state representations rather than resending full dialogues. Automatic prompt compaction and constrained templates keep input tokens low with minimal quality loss when designed carefully.

Worked example: replace full dialogue history with a one-line state vector plus the last two turns. Or summarise the previous exchange into a 50-token summary that the model can use instead of 1,000 tokens of raw history.

7. Consider model compression for self-hosted and open models

Seventh, when you self-host or run open models, apply compression techniques such as quantization, pruning, distillation and early stopping. Techniques like 8-bit quantization and knowledge distillation shrink model footprints and lower inference cost. Post-training quantization and quantization-aware training have different accuracy trade-offs, so A/B test quality after each change.

Distilled student models can approximate teacher capabilities at a fraction of the cost for many tasks. Use these methods where latency and accuracy constraints align with lower precision inference, and maintain rollbacks if accuracy degrades.

8. Protect experience with confidence-based escalation

Eighth, build routing fallbacks and confidence-based escalation to protect user experience. Put in place a two-stage flow where a low-cost model attempts a generation and a higher-cost model is used only on low-confidence outputs or when ground-truth accuracy is required. Measure failure modes and the rate of escalation because excessive escalations will erase savings.

Worked example: a cheap model produces an answer and emits a confidence score. If confidence is below the SLA threshold, escalate to a more capable model. Track the proportion of escalations and the business cost of any incorrect low-cost responses.

9. Instrument FinOps and feature-level attribution

Ninth, make FinOps visible. Build a dashboard that surfaces cost per 1K tokens, cost per request, cost by feature, p95 latency and share of traffic by model tier. Track transition costs such as engineering time and model maintenance. Compare baseline run-rate to optimized run-rate using a simple formula that subtracts transition costs from expected monthly savings so decision-makers can prioritise investments.

Worked example: a FinOps dashboard shows Feature A cost 60 percent of the model bill. After caching and routing, Feature A cost drops 70 percent. Subtract engineering and maintenance to compute net monthly saving and the payback period for the change.

10. Design governance and deletion for Canadian privacy rules

Tenth, build data governance, deletion and traceability into optimisations to meet Canadian privacy obligations. Provincial privacy rules require demonstrable traceability of personal data flows, the ability to comply with erasure requests and, in some cases, completion of Privacy Impact Assessments before deployments that handle residents' personal information. Vector embeddings, cached prompts and fine-tuning datasets all create persistence points for personal data that must be locatable and removable.

Map where customer data flows through retrieval systems, caches and any third-party inference endpoint. Adopt architectures that separate personal data from RAG indexes or permit targeted removal of vectors and cache entries. For cached items or vector indices that contain PII, build deletion and audit mechanisms that permit targeted erasure without full retraining.

11. Sequence by return on investment

Eleventh, follow a sequence that maximises ROI. Measure first, then implement prompt caching and precise cache invalidation policies because they offer the highest return with the lowest quality risk. Next deploy model routing and right-sizing to convert per-request savings into sustained run-rate reduction. Use batching and offline processing where acceptable. Finally, apply compression and distillation for incremental savings. Maintain monitoring and rollback paths at each stage.

Worked example: a programme that starts with caching and trimming prompts will typically see quick wins within weeks. Routing and batching deliver the steady-state monthly savings. Compression is the final, engineering-heavy step with incremental gains.

12. Plan for variation and test continuously

Twelfth, expect variation by workload and plan trade-offs. Reported outcomes differ by application. HighPeakSW frames a practical 30-60 percent reduction from a combination of batch, caching and routing, while practitioner posts and multi-strategy guides report higher upside ranges such as 50-90 percent or single-case reductions up to about 90 percent when multiple levers align and traffic patterns are favourable. Quantitative results hinge on how much input token volume is repeated, the share of requests that can be served by smaller models and how many non-urgent jobs you can batch.

Thirteenth, test quality impact continuously. Any change that compresses or substitutes models requires live experiments and accuracy metrics tied to business KPIs. Run A/B tests, monitor user satisfaction and instrument escalation rates from low-cost paths to high-cost models. Quantify the cost of any quality regressions against savings to make objective trade-offs.

Worked example: run a controlled A/B test where 10 percent of traffic follows the new routing rules. Monitor customer satisfaction scores, escalation rate and net cost impact. If escalation exceeds a pre-set threshold, pause and refine the classifier.

13. Canadian checklist before production

Fourteenth, complete the local checklist before routing production traffic through third-party inference. Catalogue whether requests involve Quebec residents' personal information. Complete or update a Privacy Impact Assessment for new deployments and significant system changes. Ensure traceability of data flows and document cross-border transfer justifications where applicable.

For cached items or vector indices that contain PII, build deletion and audit mechanisms that permit targeted erasure. Prioritise prompt caching, model routing and right-sizing as the first technical changes that yield the highest cost reduction per unit of engineering effort. Complete the Privacy Impact Assessment and data-flow mapping before deploying or making significant changes to an LLM stack that processes residents' personal information.

Worked example: before shifting routing logic to a new cloud region, document where each piece of PII travels, where vectors are stored and how a deletion request will be executed. That documentation helps meet provincial privacy duties and prevents regulatory surprises.

Finally, remember that the single-most-critical metric is baseline input-token volume and repeatability. If much of your input volume is repeated, caching and compaction will be the dominant lever. If most requests are unique or high-risk, routing and batching will matter more. Use the measurement rig you built in step one to decide.

Related Articles

Complete the Privacy Impact Assessment and data-flow mapping before deploying or making significant changes to an LLM stack that processes residents' personal information. Start with prompt caching, model routing and right-sizing, they deliver the largest cost reduction per unit of engineering effort.

This article was created with AI assistance.