In the first post, we looked at transformers, token prediction, training, inference, and model parameters. That gives us the machinery. Now we can ask a more practical question: how do people actually use LLMs well?
This post focuses on the layer between the model and the user: prompting, context windows, retrieval-augmented generation, tool use, and hallucinations. These ideas matter because the raw model is only part of an LLM system. The prompt, surrounding context, retrieved evidence, decoding settings, and external tools all shape the final answer.
1. Prompting Is Interface Design
A prompt is not just a question. It is the model’s immediate operating environment. It can include the user’s request, system instructions, examples, formatting rules, retrieved documents, tool outputs, and prior conversation.
For simple tasks, a short prompt can be enough. For harder tasks, good prompting makes the task explicit:
This kind of prompt does four useful things. It states the task, names the audience, defines the output format, and places a boundary around unsupported claims. It does not make the model perfect, but it reduces ambiguity.
Most API-based systems structure the prompt as a list of messages with designated roles. Here is a typical example:
# Typical API prompt structure
messages = [
{"role": "system", "content": "You are a transportation research assistant. "
"Cite sources. Say when evidence is missing."},
{"role": "user", "content": "Summarize the key factors in household "
"evacuation decision-making."}
]
The system message sets the model’s behavior for the session. The user message carries the actual request. Some APIs also support an assistant role for multi-turn conversation history, and a tool role for returning results from function calls.
Who should the model write for?
What should it actually do?
What sources may it use?
What should the answer look like?
What should it avoid?
How should uncertainty be handled?
- Brown et al. (2020), "Language Models are Few-Shot Learners": showed how examples inside the prompt can steer model behavior without updating weights.
- Wei et al. (2022), "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models": introduced a major prompting pattern for eliciting intermediate reasoning on multi-step tasks.
- Kojima et al. (2022), "Large Language Models are Zero-Shot Reasoners": showed that simple prompt phrasing can improve reasoning behavior without examples.
2. In-Context Learning Is Not Fine-Tuning
When you include examples in a prompt, the model may infer the pattern and continue it. This is called in-context learning. The model’s weights are not changed. The examples are temporary context.
Fine-tuning is different. Fine-tuning updates model weights using training data. It is useful when you need durable behavior across many future prompts, a specialized style, or domain-specific output conventions.
Examples are placed in the prompt.
No parameter update happens.
Good for rapid adaptation and prototypes.
Examples become training data.
Parameters are updated.
Good for durable task behavior.
The practical rule is simple: try prompting first, add retrieval when factual grounding matters, and consider fine-tuning when the same behavior must repeat reliably at scale.
- Brown et al. (2020), "Language Models are Few-Shot Learners": the central reference for few-shot prompting and in-context learning in GPT-style models.
- Wei et al. (2021), "Finetuned Language Models Are Zero-Shot Learners": the FLAN paper, important for instruction tuning across many tasks.
- Ouyang et al. (2022), "Training Language Models to Follow Instructions with Human Feedback": explains why post-training can make models more helpful and instruction-following.
3. Context Windows Are Working Memory
The context window is the amount of text the model can consider at once. It includes system instructions, user messages, previous turns, retrieved documents, tool results, and generated output.
Longer context is useful, but it is not magic. A model may technically accept a long document while still missing important information in the middle, mixing unrelated sections, or over-weighting the most recent text.
System instructions
Conversation history
Retrieved documents
Space for answer
These are illustrative slices, not a fixed formula.
Good context design is selective. Put the most relevant evidence close to the question. Remove stale conversation when it no longer helps. Summarize long histories. Do not paste a giant pile of documents and hope the model sorts everything out.
- Liu et al. (2023), "Lost in the Middle": shows that models often use information better when it appears near the beginning or end of context than in the middle.
- Dao et al. (2022), "FlashAttention": important background for why efficient attention matters when context length grows.
4. Retrieval-Augmented Generation
Retrieval-Augmented Generation, usually called RAG, connects a language model to an external knowledge source. Instead of relying only on what is stored in the model’s parameters, the system retrieves relevant text and places it into the prompt.
The user asks something.
Search finds relevant chunks.
The system chooses useful evidence.
The LLM answers using context.
The answer points back to sources.
RAG is especially useful when the answer depends on current, private, local, or domain-specific information. A model may know general evacuation planning concepts, but your project documents, survey instruments, data tables, and meeting notes need retrieval.
RAG also introduces new failure modes. The retriever can miss the right document. The ranking step can include irrelevant chunks. The model can ignore evidence or cite the wrong passage. A good RAG system therefore needs evaluation, not just a vector database.
- Lewis et al. (2020), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks": the core RAG paper that combines parametric generation with non-parametric retrieval.
- Guu et al. (2020), "REALM": an important retrieval-augmented pretraining paper for open-domain question answering.
- Gao et al. (2023), "Retrieval-Augmented Generation for Large Language Models: A Survey": a broad survey of naive, advanced, and modular RAG systems.
5. Tools Extend the Model
Some tasks should not be solved by language generation alone. Arithmetic, database queries, web search, code execution, scheduling, mapping, and file operations are better handled by tools.
In a tool-using system, the model decides when to call an external function, passes arguments, receives the result, and then continues generating. The model is still producing text, but the system gives it ways to act.
The point is not to make the model sound smarter. The point is to move fragile work into systems that can be checked.
- Yao et al. (2022), "ReAct": combines reasoning traces with actions such as searching or interacting with an environment.
- Schick et al. (2023), "Toolformer": explores how language models can learn to call APIs and incorporate tool results.
- Gao et al. (2022), "PAL: Program-aided Language Models": shows how LLMs can write programs as intermediate steps and let a runtime compute the answer.
6. Structured Output and Function Calling
Many real applications need the model to produce output in a specific format rather than free-form prose. The target might be JSON, a function call, a database query, a set of form fields, or a structured data record. This matters because downstream systems — APIs, databases, user interfaces — need to parse the model’s response reliably. Free-text answers that mix data with explanation are hard to consume programmatically.
Function calling is one common pattern. The model does not execute code itself. Instead, it produces a structured object describing which function to call and what arguments to pass. The system then executes the function and returns the result.
Model response (function call): { “function”: “query_traffic_database”, “arguments”: { “route”: “US-50”, “location”: “bridge_segment_3”, “metric”: “average_daily_traffic” } }
The system receives this JSON, calls the actual database function, and feeds the result back to the model for a final answer. This keeps the model in the role of deciding what to ask for, while a reliable system handles how to get it.
Important caveat: structured output reduces ambiguity but does not guarantee correctness. The model can produce well-formed JSON that contains wrong values — a valid function call with the wrong route number, the wrong metric name, or a hallucinated location identifier. Always validate structured outputs against your schema and, where possible, against the actual data.
Structured output is increasingly supported directly by model APIs through mechanisms like JSON mode, function definitions, and response format schemas. These constraints help ensure the output parses correctly, but they do not solve the deeper problem of whether the content is right.
7. Hallucinations Are Grounding Failures
A hallucination is generated content that sounds plausible but is unsupported, false, or disconnected from the available evidence. This is not just a “bug” in the conversational sense. It follows naturally from the training objective: the model is rewarded for likely continuations, not for independently verifying reality.
Hallucinations often appear when:
- the prompt asks for information outside the model’s knowledge;
- the user requests citations the model has not actually retrieved;
- the context contains conflicting evidence;
- the model is asked to infer too much from too little;
- the output format rewards confidence over uncertainty.
Practical mitigation: give the model evidence, ask it to separate facts from assumptions, require citations for factual claims, and make "I do not know" an acceptable answer.
RAG, tools, and careful prompting can reduce hallucinations, but they do not eliminate them. The more important habit is to treat the LLM as a probabilistic assistant inside a verification workflow.
- Lin, Hilton, and Evans (2021), "TruthfulQA": a benchmark for measuring whether models reproduce common falsehoods.
- Manakul, Liusie, and Gales (2023), "SelfCheckGPT": a black-box approach to detecting hallucinations by checking consistency across sampled outputs.
- Bender et al. (2021), "On the Dangers of Stochastic Parrots": an influential critique of risks around scale, data, bias, and over-attributing understanding to fluent text.
8. A Practical Workflow
For many real tasks, I would think about LLM use in this order:
- Clarify the task. What exactly should the model produce?
- Provide context. What does the model need to know for this answer?
- Ground factual claims. Should this use retrieved sources, data, or tools?
- Constrain the output. What format, tone, and level of detail are useful?
- Ask for uncertainty. Where should the model flag missing or weak evidence?
- Verify. What parts need human review, tests, citations, or computation?
The model is not the whole system. The system is the model plus instructions, context, retrieval, tools, interface design, evaluation, and human judgment.
In the next post, we can go deeper into RAG: embeddings, chunking, vector search, reranking, citations, and evaluation.
- Zhao et al. (2023), "A Survey of Large Language Models": useful as a broad map of model architecture, training, adaptation, use, and evaluation.
- Gao et al. (2023), "Retrieval-Augmented Generation for Large Language Models: A Survey": a good next step if you want to understand practical RAG design space.
- You paste a 50-page report into a prompt. The model gives a confident but wrong answer about content on page 25. Using what you know about context windows, list two possible explanations.
- A colleague adds three examples of the desired output format into the prompt. The model starts producing similar outputs. Is the model fine-tuned? Explain the difference between in-context learning and fine-tuning.
- You build a RAG system and the model returns an answer with a citation. How would you verify that the citation actually supports the claim? What could go wrong?