LLM 107: Deployment, Cost, Latency, and Monitoring

In LLM 106, we looked at fine-tuning and adaptation. That gave us a way to change model behavior when prompting, retrieval, and tools are not enough. Now we move from model behavior to system behavior.

An LLM prototype can be a notebook, a prompt, and a few examples. A deployed LLM system is different. It has users, latency expectations, costs, logs, privacy constraints, model updates, failures, and maintenance work. This post focuses on the practical side: model choice, inference cost, serving, caching, monitoring, privacy, and operational reliability.

1. Deployment Is a System Problem

The model is only one component. A deployed LLM application usually includes prompts, retrieval, tools, routing, user interface, logging, monitoring, safety checks, access control, and evaluation.

The core deployment question is not “Can the model answer this once?” It is “Can the system answer this reliably, affordably, and safely for real users?”

2. Choosing a Model

Model choice is a tradeoff. Larger models often perform better on difficult reasoning and broad knowledge tasks, but they cost more and are slower. Smaller models can be cheaper, faster, easier to host, and good enough for narrow workflows.

For many applications, the best system uses more than one model. A small model might classify intent, a retrieval system might gather evidence, and a stronger model might synthesize the final answer.

Important model-choice questions:

• What task must the model perform?
• Does it need tool use or structured output?
• How much context does it need?
• What is the acceptable latency?
• What is the cost per successful task, not just cost per token?
• Can data leave your environment?
• How will you evaluate regressions after model changes?

For example, a hurricane-season planning assistant that processes confidential municipal evacuation routes might require an open-weight model hosted on-premises to satisfy data governance requirements.

3. Open vs Closed Models

Not all models are accessed the same way. Closed models (e.g., GPT-4, Claude) are accessed through APIs. You send your data to the provider’s servers and receive completions back. Open-weight models (e.g., Llama, Mistral, Gemma) publish their weights so you can download them and run inference on your own infrastructure.

This distinction affects cost structure, data privacy, customization options, and long-term reliability.

4. Latency: Time to First Token and Time to Done

LLM latency has two parts. Time to first token is how long the user waits before seeing the answer begin. Time to completion is how long the full answer takes.

Long prompts increase prefill time. Long answers increase decoding time. Tool calls, retrieval, reranking, and safety checks add their own latency.

Useful latency techniques include:

• streaming tokens to the user;
• reducing prompt size;
• retrieving fewer but better chunks;
• caching repeated prompts or retrieval results;
• using smaller models for simple tasks;
• batching requests when throughput matters;
• limiting output length;
• using faster decoding or serving systems.

5. Cost Is More Than Tokens

Token cost matters, but it is not the whole bill. A deployed system may pay for model calls, embedding calls, vector databases, rerankers, tool execution, storage, monitoring, human review, and engineering maintenance.

Think in terms of cost per successful task:

A cheap model that fails often can be more expensive than a stronger model that succeeds with fewer retries. A long prompt that avoids one tool call may still cost more than a short prompt with a deterministic tool.

If a transportation planning office uses an LLM assistant to process 200 evacuation scenario reports per season, the cost-per-task calculation should include the full pipeline: document retrieval, reranking, generation, and human review of each output.

6. Serving and the KV Cache

During generation, transformer models store key-value representations for previous tokens. This KV cache avoids recomputing the entire history at every step. It is essential for efficient decoding, but it can use a lot of memory.

Serving systems care about:

• how many requests can run at once;
• how long each request is;
• how much KV cache memory is needed;
• whether requests can be batched;
• whether shared prefixes can reuse computation;
• how to schedule work under load.

7. Quantization and Compression

Quantization reduces the precision used to store or compute with model weights. The goal is to reduce memory and sometimes increase speed while keeping quality acceptable.

This is especially important when deploying models on limited hardware. But quantization is a tradeoff. Lower precision can reduce quality, affect rare tasks, or interact badly with certain model architectures.

Always evaluate the quantized model on your actual task. A quantized model may look fine on casual chat but fail on structured extraction, citations, math, or domain-specific wording.

8. Caching

Caching can reduce cost and latency, but only when used carefully.

Useful cache targets include:

• embeddings for documents;
• retrieval results for repeated queries;
• reranking results;
• prompt prefixes;
• tool outputs;
• final answers for deterministic or low-risk tasks.

Caching is less safe when answers depend on user permissions, current data, or private context. If two users ask the same question but have different document access, they should not automatically receive the same cached answer.

9. Monitoring Quality in Production

You cannot manually inspect every output. You need monitoring signals.

Track:

• latency;
• cost;
• token counts;
• retry rate;
• tool-call failure rate;
• retrieval hit rate;
• citation coverage;
• user corrections;
• escalation rate;
• refusal rate;
• human review findings.

Monitoring should connect back to evaluation. If your evaluation says citation quality matters, production monitoring should track citation failures. If your agent has tools, monitor tool calls and not only final text.

10. Privacy, Logging, and Data Boundaries

LLM systems often touch sensitive data: documents, prompts, user messages, tool outputs, identifiers, and metadata. Deployment requires explicit data boundaries.

Questions to answer before launch:

• What data is sent to the model provider?
• What is logged?
• How long are logs retained?
• Can logs contain private documents or personal information?
• Who can inspect traces?
• Are retrieved documents filtered by user permissions?
• Can users request deletion or correction?
• Are prompts and outputs used for future training?

Privacy is not a layer you add at the end. It affects model choice, hosting choice, retrieval permissions, logging, monitoring, and review workflows.

11. Rollouts and Regression Testing

LLM systems should be rolled out like software systems, not like one-off demos.

Use:

• staging environments;
• fixed evaluation sets;
• canary releases;
• prompt and model versioning;
• rollback plans;
• human review for high-risk outputs;
• monitoring dashboards;
• incident notes when failures occur.

The boring versioning details matter. If quality gets worse, you need to know whether the model changed, the prompt changed, the retriever changed, the documents changed, or user behavior changed.

12. A Practical Deployment Checklist

Before shipping an LLM feature, I would check:

1. Task definition: What workflow does this support?
2. Model choice: Why this model, and what is the fallback?
3. Prompt versioning: Can you reproduce a prior output?
4. Context design: What enters the prompt, and why?
5. Cost controls: Are there token, retry, and tool-call budgets?
6. Latency controls: Is streaming, caching, or routing needed?
7. Privacy rules: What data is sent, stored, redacted, or blocked?
8. Evaluation: What local tests must pass before release?
9. Monitoring: What signals show quality, risk, and cost?
10. Rollback: How do you revert a bad model, prompt, or index update?

In the next post, we can look at safety and responsible use: bias, privacy, misuse, transparency, uncertainty, and human oversight.