Local LLMs on Mohammad Movahedi

Local LLMs 209: A Practical Local Agent Stack

Fri, 05 Jun 2026 00:00:00 +0000

After the tools, models, routers, agents, skills, and MCP servers, it helps to put the pieces into one practical stack.

The goal is not to build the most complicated local AI system. The goal is to build a workflow you can understand, debug, and improve.

Mental model: start local, add power only when needed

A practical beginner stack has layers:

Layer	Default choice	Why
Runtime	Ollama or another beginner-friendly local server	Gets a model running quickly
Model	Small or medium instruct model	Establishes a stable baseline
API shape	OpenAI-compatible endpoint when possible	Works with many clients
Evaluation	Fixed prompt set	Prevents vibe-based model choice
Tools	MCP servers or app-native tools	Adds controlled access to context
Instructions	Rules, skills, or project notes	Makes behavior repeatable
Hosted fallback	OpenRouter or direct provider API	Handles tasks local models cannot
Coding agent	Claude Code, Codex, Antigravity, or another tool	Handles repository actions and verification

The Practical Local Agent Stack

Click a layer to see when to add it

Local LLMs 208: MCP and How Local LLMs Get Tools and Context

Thu, 04 Jun 2026 00:00:00 +0000

A local model by itself only sees the prompt you give it. It cannot read your files, inspect GitHub issues, search a database, or check a calendar unless something connects it to those sources.

MCP, the Model Context Protocol, is one way to standardize that connection.

Mental model: MCP is a tool plug

Anthropic’s MCP documentation describes MCP as an open protocol that standardizes how applications provide context to LLMs. The common analogy is a standard connector between AI applications and data or tools.

Local LLMs 207: Skills, Rules, Agents, and Subagents

Wed, 03 Jun 2026 00:00:00 +0000

Once you use LLM tools for more than casual chat, you run into a problem: you keep repeating the same instructions. “Use this style.” “Check these files.” “Run this test.” “Prefer this source.” “Do not edit generated files.”

Skills, rules, agents, and subagents are different ways of making those instructions reusable.

Mental model: durable context for repeatable work

Think of these concepts as layers of reusable behavior:

Concept	Beginner meaning
Rule	A standing instruction or constraint
Skill	A reusable workflow with instructions, references, scripts, or assets
Agent	A model-driven worker with tools and a task
Subagent	A specialized helper agent delegated by a main agent
Project instructions	Repo-specific guidance for how work should be done
MCP tool	A live capability exposed to a model, such as reading issues or querying a database

Durable Context Hierarchy

Click an entity to see its role

Local LLMs 206: Coding Agents

Tue, 02 Jun 2026 00:00:00 +0000

A chat model answers questions. A coding agent works inside a software environment. That difference sounds small until the agent starts reading files, editing code, running tests, opening browsers, creating commits, and explaining what changed.

Claude Code, Codex, and Google Antigravity are examples of agentic coding tools. They do not replace local LLM runtimes, but they show what happens when a model is wrapped in tools, permissions, project context, and verification loops.

Local LLMs 205: Hosted Routers and BYOK: OpenRouter.ai in the Local LLM World

Mon, 01 Jun 2026 00:00:00 +0000

Local LLMs are about control. Hosted routers are about reach. At first those sound opposite, but many practical workflows use both.

OpenRouter is a hosted service that provides access to many models through one API. That makes it useful when you want to compare models, keep one client interface, or fall back to a stronger hosted model when your local machine is not enough.

Mental model: local-first, hosted fallback

A simple hybrid workflow looks like this:

Local LLMs 204: Why the Same Model Is Not Always the Same

Sun, 31 May 2026 00:00:00 +0000

You may see the same model name in Ollama, Hugging Face, LM Studio, OpenRouter, a benchmark table, and a coding tool. Then you try it in two places and it behaves differently.

This is normal. “Same model” often hides several layers of difference.

Mental model: name, checkpoint, format, runtime, provider

A model name is only the label at the top of the stack. Behavior comes from the whole stack:

Layer	Example difference
Family	Qwen, Llama, Gemma, Mistral, DeepSeek, Phi
Checkpoint	Base, instruct, coder, reasoning, distilled, updated release
File format	GGUF, safetensors, runtime-specific package
Quantization	Q4, Q5, Q8, full precision
Runtime	Ollama, llama.cpp, vLLM, MLX, transformers
Prompt template	ChatML, model-specific instruct format, custom system prompt
Provider	Local machine, OpenRouter route, direct vendor API
Parameters	Temperature, top_p, max tokens, reasoning settings, stop sequences

The "Same Model" Stack

A model name is just the top layer.

Local LLMs 203: Choosing a Model Without Guessing

Sat, 30 May 2026 00:00:00 +0000

Local model choice can turn into name collecting very quickly. Llama, Qwen, Gemma, Mistral, DeepSeek, Phi, gpt-oss, coder models, reasoning models, small models, giant models, quantized models, preview models: it is a lot.

The goal is not to memorize every release. The goal is to choose a model with a reason.

Mental model: model choice is task fit plus hardware fit

A useful model is the intersection of three things:

Local LLMs 202: The Local LLM Toolchain: uv, uvx, Ollama, and Model Files

Fri, 29 May 2026 00:00:00 +0000

A local LLM setup can feel messy because several tools appear at once: Python environments, command-line utilities, model downloaders, model runtimes, chat interfaces, and APIs. The trick is to separate the toolchain into jobs.

For a beginner, four ideas are enough to get oriented: uv, uvx, Ollama, and model files.

Mental model: tools install things, runtimes run models

uv helps manage Python projects, Python versions, dependencies, and Python command-line tools. Ollama runs language models and exposes them through a local API. A model file contains the model weights and metadata that the runtime needs.

Local LLMs 201: What Runs on Your Machine

Thu, 28 May 2026 00:00:00 +0000

Local large language models are exciting because they move part of the AI stack onto your own computer. Instead of sending every prompt to a remote service, you download a model, run it through a local runtime, and talk to it from an app, terminal, editor, or API.

The beginner mistake is to ask, “What is the best local model?” The better first question is: what can this machine actually run well enough to be useful?