LLM 106: Fine-Tuning, Adaptation, and When to Train

In LLM 105, we focused on evaluation: how to measure whether an LLM system is useful, faithful, and reliable. Evaluation naturally leads to adaptation. Once you know where a model fails, you can decide how to improve it.

The tempting answer is often “fine-tune it.” Sometimes that is right. Often it is not. Many LLM problems are better solved with a clearer prompt, better retrieval, a tool call, stronger evaluation, or a narrower workflow. Fine-tuning is powerful, but it is not magic. It changes model behavior; it does not automatically add fresh facts, guarantee truthfulness, or replace system design.

1. Adaptation Is a Spectrum

There are many ways to adapt an LLM system. Fine-tuning is only one of them.

The right choice depends on the failure mode. If the model lacks current facts, use retrieval. If it does arithmetic poorly, use a calculator or code tool. If it writes in the wrong format every time, fine-tuning may help. If it gives answers users dislike even when they are technically valid, preference tuning may help.

2. Prompting vs RAG vs Fine-Tuning

These methods solve different problems.

Suppose you are building a research assistant for evacuation papers.

• If it summarizes too verbosely, adjust the prompt.
• If it misses project-specific documents, improve RAG.
• If it consistently fails to follow your lab’s structured review template, consider fine-tuning.
• If it invents statistics, add evidence requirements and evaluation before training.

3. Supervised Fine-Tuning

Supervised fine-tuning (SFT) trains the model on input-output examples. The model learns to produce outputs that look like the target examples.

SFT is useful for formats, domain style, recurring workflows, and task conventions. It is less useful for facts that change. If you fine-tune on old policy documents, the model may internalize outdated information. A retriever is usually a better home for facts.

Good SFT data has:

• consistent instructions;
• high-quality target outputs;
• realistic examples;
• edge cases;
• refusals or uncertainty cases;
• clear formatting;
• held-out evaluation examples.

4. Full Fine-Tuning vs Parameter-Efficient Fine-Tuning

Full fine-tuning updates all model parameters. This can be powerful, but it is expensive and can require storing a full copy of the adapted model.

Parameter-efficient fine-tuning (PEFT) updates only a small number of additional parameters or selected components. The base model stays mostly frozen.

LoRA is one of the most influential PEFT methods. Instead of changing a large weight matrix directly, it learns small low-rank updates. QLoRA goes further by fine-tuning adapters while the base model is quantized, reducing memory requirements.

Here is a typical LoRA configuration to give you a sense of the moving parts:

# Example: LoRA configuration (using Hugging Face PEFT)
from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                   # rank of the low-rank matrices
    lora_alpha=32,          # scaling factor
    target_modules=[        # which layers to adapt
        "q_proj", "v_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
# This adds ~0.1% new parameters while keeping
# the base model frozen.

5. Distillation: Learning from a Larger Model

Distillation trains a smaller model (the student) to imitate the outputs of a larger model (the teacher). Instead of training the student from scratch on raw data, you use the teacher’s probability distributions or generated outputs as the training signal. The student learns not just the correct answer, but the teacher’s confidence and reasoning patterns.

Why does this matter? You get a smaller, faster, cheaper model that retains much of the teacher’s capability for specific tasks. A 70B-parameter model may be too expensive to serve at scale, but a 7B student distilled from it can handle a well-defined task surprisingly well at a fraction of the cost.

6. Preference Tuning

Sometimes you do not only want the model to imitate examples. You want it to prefer one kind of answer over another.

Preference data usually contains pairs:

The chosen answer is not just a correct answer; it better matches the desired behavior. Preference tuning methods use these comparisons to shape model outputs.

RLHF uses reward modeling and reinforcement learning. DPO offers a simpler direct optimization route using preference pairs. Constitutional AI explores using written principles and AI feedback to critique and revise outputs.

7. Data Quality Is the Whole Game

Fine-tuning is not mainly a modeling problem. For most teams, it is a data problem.

Bad training data can teach:

• wrong facts;
• inconsistent formatting;
• shallow reasoning;
• unsupported citations;
• overconfident answers;
• hidden bias;
• brittle refusal behavior;
• leakage of private information.

Before training, ask:

1. Who wrote the examples?
2. Are they correct?
3. Are they representative of real use?
4. Do they include difficult cases?
5. Are sources and permissions clean?
6. Is there a separate evaluation set?
7. What behavior should the model not learn?

8. Evaluation Before and After Training

Fine-tuning without evaluation is vibes with a GPU bill.

Use the evaluation ideas from LLM 105 before training:

• create a local evaluation set;
• define success metrics;
• include edge cases;
• separate factual grounding from style;
• test baseline prompting and RAG first;
• compare before and after fine-tuning;
• check regressions on general behavior.

If fine-tuning improves format but harms truthfulness, you have not solved the problem. You have moved it. The strongest adaptation workflows evaluate both the target task and the behaviors you do not want to lose.

9. When Fine-Tuning Is the Right Move

Fine-tuning is often worth considering when:

• the same task is repeated many times;
• the desired format is stable;
• the model needs a domain-specific style;
• high-quality training examples exist;
• prompting becomes too long or fragile;
• RAG provides facts but the model still mishandles the task;
• latency or cost improves by replacing long prompts with learned behavior;
• evaluation shows a stable failure pattern.

For example, a transportation research lab might fine-tune a model to produce a standard paper-review template. The facts still come from retrieved papers. The fine-tuning teaches the model the lab’s review structure, level of detail, and cautious phrasing.

10. When Fine-Tuning Is the Wrong Move

Fine-tuning is probably the wrong first move when:

• the facts change frequently;
• the problem is missing retrieval;
• the failure is caused by unclear instructions;
• you do not have good examples;
• the task requires exact computation;
• the risk is from unsupported claims;
• the workflow needs access control or citations;
• you cannot evaluate the result.

11. A Practical Adaptation Checklist

Before fine-tuning, I would walk through this checklist:

1. Name the failure. Is it style, format, reasoning, grounding, tool use, or factual knowledge?
2. Try the simpler fix. Prompt, RAG, tool, schema, or evaluation change.
3. Collect examples. Use realistic inputs and high-quality target outputs.
4. Hold out eval data. Do not train on the examples you need for testing.
5. Choose the tuning method. SFT, LoRA, QLoRA, DPO, or another preference method.
6. Train narrowly. Avoid broad behavior changes unless you can evaluate them.
7. Compare to baseline. Measure actual improvement, not just nicer demos.
8. Check regressions. Truthfulness, safety, formatting, latency, cost, and refusal behavior.
9. Keep facts external. Use RAG for updateable knowledge.

In the next post, we can look at deployment: latency, cost, model choice, caching, monitoring, privacy, and how to make an LLM system behave well outside a notebook.