LLM 105: Evaluation, Benchmarks, and Measuring Usefulness

In LLM 104, we looked at agents, tools, memory, and guardrails. That raises the question every serious LLM system eventually faces: how do we know if it is actually good?

Evaluation is harder than it looks. A model can sound fluent while being wrong. A RAG system can cite sources while missing the best evidence. An agent can complete a task but take too many steps, use the wrong tools, or create avoidable risk. Good evaluation is not just a leaderboard score. It is a measurement plan tied to the task you care about.

1. Evaluation Starts with the Use Case

The first mistake is asking, “Which model is best?” before asking, “Best for what?”

For a research assistant, quality might mean faithful summaries with citations. For a coding assistant, it might mean passing tests. For a customer support bot, it might mean resolving issues without escalation. For an evacuation planning assistant, it might mean correctly extracting assumptions, constraints, data sources, and uncertainty from planning documents.

An evaluation should be built around representative tasks, not just abstract prompts. If the actual workflow involves messy PDFs, incomplete tables, acronyms, and domain-specific definitions, the evaluation should include those.

2. Benchmarks Are Useful but Incomplete

Benchmarks give us shared measurement. They make it possible to compare models on common tasks. But benchmarks can also be overfit, contaminated, narrow, or disconnected from your actual use case.

This is why benchmark results should be treated as a starting point. They tell you which models may be worth testing, not which system will work in your domain.

3. Build a Local Evaluation Set

For a real LLM application, a small local evaluation set is gold. It should contain examples that look like the work users actually do.

For a research or planning assistant, examples might include:

• summarizing one paper with citations;
• comparing two methods sections;
• extracting variables from a survey instrument;
• answering a question from a long report;
• identifying missing evidence;
• refusing to invent numbers;
• using a tool to compute a simple result;
• catching a contradiction between two sources.

You do not need hundreds of examples to start. A carefully designed set of 30-50 examples can reveal much more than a pile of generic prompts.

4. Human Evaluation Still Matters

Humans are expensive, inconsistent, and slow. They are also still essential. Some qualities cannot be fully reduced to automatic metrics: usefulness, clarity, nuance, trust, appropriateness, and whether the answer helped the user think.

A simple human rubric can be more useful than a complicated score:

Human evaluation becomes stronger when reviewers see clear criteria, examples of each score level, and instructions for resolving uncertainty.

5. LLM-as-a-Judge

One popular approach is using a stronger LLM to evaluate model outputs. This can scale evaluation quickly, especially for drafts, summaries, and conversation quality. But an LLM judge is still a model. It can be biased toward longer answers, familiar phrasing, confident style, or outputs from models it recognizes.

Use LLM judges as part of the evaluation stack, not as the entire stack.

Good practice includes blind comparisons, short rubrics, checking judge agreement with humans, and periodically reviewing judge failures.

6. Evaluating RAG Systems

We introduced RAG evaluation concepts in LLM 103 (sections 8 and 9). Here we connect those ideas to a broader measurement framework.

RAG evaluation needs two separate questions:

1. Did retrieval find the right evidence?
2. Did generation use that evidence faithfully?

If the answer is wrong, you need to know where the failure happened. Did the retriever miss the source? Did the reranker bury it? Did the model ignore it? Did the prompt ask for more certainty than the evidence supports?

For a RAG assistant, I would test questions where the answer is present, absent, ambiguous, contradicted across sources, or spread across multiple documents. The “absent” cases are especially important because they test whether the system can avoid bluffing.

7. Evaluating Agents

Agents require process evaluation. The final answer may look fine while the agent took unnecessary actions, ignored tool output, or got lucky after a bad plan.

Track:

• task success;
• tool-call correctness;
• number of steps;
• cost and latency;
• recovery from errors;
• permission boundary violations;
• human interventions;
• auditability of the trace.

The baseline matters. If a deterministic script plus one LLM call solves the task more reliably than an agent loop, the simpler system wins.

8. Regression Testing

LLM systems change. Models update, prompts change, retrieval indexes refresh, documents move, tools break, and user expectations evolve. Regression testing helps catch behavior that quietly got worse.

Useful regression tests include:

• fixed prompts with expected traits;
• RAG questions with known supporting chunks;
• tool-use tasks with expected calls;
• refusal or uncertainty cases;
• citation checks;
• cost and latency thresholds;
• adversarial or malformed inputs.

9. A Practical Evaluation Plan

For a serious but manageable LLM evaluation, I would start here:

1. Define the task. Write down what the system is supposed to do.
2. List failure modes. Hallucination, bad citations, missed retrieval, wrong tool calls, unsafe actions.
3. Create 30-50 local examples. Include normal cases and edge cases.
4. Write a rubric. Keep it short enough that reviewers actually use it.
5. Separate components. Evaluate retrieval, generation, and tool use independently.
6. Run human review. Sample outputs and calibrate the rubric.
7. Add automated checks. Use exact tests where possible and LLM judges where helpful.
8. Track cost and latency. A better answer that is too slow or expensive may not be useful.
9. Repeat after changes. Treat evaluation as part of development.

In the next post, we can look at fine-tuning and adaptation: when prompting is enough, when RAG is enough, and when it makes sense to update model behavior with training data.