LLM 101: Transformers, Training, and Inference

Large Language Models (LLMs) can look mysterious from the outside: you type a prompt, the model writes back, and the response often feels more like conversation than computation. Under the hood, though, the core idea is surprisingly concrete. Most modern LLMs are built from transformers, trained to predict text one token at a time, and used during inference to repeatedly choose the next likely token.

This post is the first in a short series. We will start with transformers, then move into how LLMs learn, how they generate text, why training and inference are different, and what people mean when they talk about model parameters.

1. Text Becomes Tokens

LLMs do not read text as whole sentences. They first break text into tokens: common words, word pieces, punctuation marks, spaces, or byte-level fragments. A sentence like this:

might become a sequence of token IDs. Each ID is then mapped into an embedding, which is a vector of numbers. You can think of an embedding as the model’s internal coordinate system for meaning, syntax, and usage.

The model also needs information about order. The same words can mean different things depending on position, so transformers add positional information to token embeddings before processing them.

# Example: tokenizing with tiktoken (used by GPT-family models)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("The evacuation plan changed.")
print(tokens)   # [976, 60038, 3197, 5614, 13]
print(len(tokens))  # 5 tokens from 4 words

2. Why Transformers Matter

Before transformers, many language models processed text mostly from left to right using recurrent neural networks. That worked, but it made long-range relationships difficult and slow to learn. Transformers changed the center of gravity by using self-attention.

Self-attention lets each token look at other tokens in the sequence and decide which ones matter for the current computation. For example, in “the bridge failed after it flooded,” the model needs to connect “it” with the right earlier concept. Attention is one of the mechanisms that helps.

Attention: The Core Equation

A Transformer Block

A transformer is built by stacking many similar blocks. Each block usually contains attention, a feed-forward network, normalization, and residual connections.

Input Tokens
    ↓
[Token Embeddings + Positional Encoding]
    ↓
┌─────────────────────────────────┐
│   Multi-Head Self-Attention     │
│   + Residual Connection         │
│   + Layer Normalization         │
├─────────────────────────────────┤
│   Feed-Forward Network          │
│   + Residual Connection         │
│   + Layer Normalization         │
└─────────────────────────────────┘
    ↓  (repeat N times)
[Final Layer Norm]
    ↓
[Linear + Softmax → Token Probabilities]

The phrase large language model usually means a transformer with many layers, wide internal vectors, and a huge number of learned weights.

3. What an LLM Learns

The most common training objective is next-token prediction. During training, the model receives text and learns to predict the next token at many positions.

For a sentence like:

During a hurricane, people may choose to evacuate because ___

the model learns to assign probabilities to possible continuations such as “roads,” “officials,” “risk,” “family,” or “flooding.” It is not choosing from a fixed list of human-written answers. It is producing a probability distribution across its vocabulary.

After seeing enormous amounts of text, the model learns patterns that support grammar, style imitation, translation, summarization, code generation, and some forms of reasoning. This is why next-token prediction is more powerful than it sounds. To predict language well, the model has to build internal representations of many things language refers to.

That said, an LLM is not a database in the ordinary sense. It stores knowledge in distributed patterns across weights, and it can be wrong, outdated, or overconfident.

4. Training Structure

During training, the model’s parameters are still changing.

1. Text is collected and cleaned.
2. Text is tokenized into long sequences.
3. The model predicts the next token at many positions.
4. The predicted probabilities are compared with the actual next tokens.
5. The error becomes a loss value.
6. Backpropagation adjusts the parameters to reduce future error.

This process is repeated over massive batches of examples. The model gradually becomes better at assigning high probability to plausible continuations and lower probability to unlikely ones.

Many deployed models also go through additional stages after pretraining, such as supervised fine-tuning, preference optimization, safety training, tool-use training, or domain adaptation. Those stages shape behavior, but the transformer foundation remains the same.

5. Inference Structure

Inference is what happens when you use a trained model. The model receives a prompt and generates output one token at a time.

The important distinction is that inference does not normally update the model’s parameters. The model may appear to learn within a conversation because previous messages remain in the context window, but that is temporary context use, not permanent weight updating.

6. Generation and Decoding

At each generation step, the model outputs probabilities for possible next tokens. The decoding strategy controls how one token is selected.

• Greedy decoding chooses the highest-probability token every time. It is predictable but can be bland or get stuck.
• Sampling randomly draws from the probability distribution. It can be more varied, but also more error-prone.
• Temperature adjusts randomness. Lower temperature makes outputs more conservative; higher temperature makes outputs more exploratory.
• Top-k sampling restricts choices to the k most likely tokens.
• Top-p sampling restricts choices to the smallest set of tokens whose combined probability passes a threshold.
• Beam search tracks several candidate continuations at once. It is useful in some structured generation tasks, though less common for open-ended chat.

This is why the same prompt can produce different answers across runs. The model is not retrieving a single stored response. It is repeatedly constructing a continuation under a probability model and a decoding policy.

7. Common LLM Architectures

Not every transformer-based language model is structured the same way. Three families matter most:

Most popular conversational LLMs are decoder-only because next-token generation maps naturally onto dialogue, writing, coding, and step-by-step responses.

8. Model Parameters

Model parameters are the learned numbers inside the network. They include embedding tables, attention projections, feed-forward layers, normalization terms, and output weights. When people say a model has 7 billion, 70 billion, or hundreds of billions of parameters, they are talking about the number of learned values used to transform input tokens into output probabilities.

More parameters can increase capacity, but parameter count is not the whole story. Data quality, architecture, training recipe, context length, tokenizer, alignment process, inference settings, and tool access all affect model behavior. A smaller well-trained model can outperform a larger poorly trained model on specific tasks.

There is also an important deployment distinction:

• Total parameters: all learned weights stored by the model.
• Active parameters: the subset used for a given token, especially relevant in mixture-of-experts models.
• Context window: the amount of text the model can consider at once. This is not the same as parameter count.

9. A Compact Mental Model

If you remember only one thing, use this:

That loop is simple. The scale is not. The power of LLMs comes from running this simple prediction game across huge datasets, with billions of learned parameters, and then using the trained model inside carefully designed inference systems.

In the next post, we can build on this foundation and look at prompting, context windows, retrieval-augmented generation, and why LLMs sometimes hallucinate.