What an LLM Is: Next-Token Prediction and the Training Pipeline

TL;DR: An LLM is one function: token sequence in, probability distribution over the next token out. Generation is that function called in a loop, appending its own output. Three training stages turn it into an assistant: pretraining (next-token prediction on trillions of tokens) buys knowledge and fluency, supervised fine-tuning teaches it to answer instead of continue, and preference alignment (RLHF or DPO) teaches it which answer a human would pick. Capability comes from pretraining; behavior comes from post-training.

The function underneath

Strip away the chat interface and an LLM does exactly one thing. It takes a sequence of tokens and returns a score for every token in its vocabulary, typically 32,000 to 200,000 entries. Softmax those scores and you have a probability distribution over what comes next. Nothing more.

Generation is that single function in a loop:

def generate(model, tokens, max_new=100):
    for _ in range(max_new):
        logits = model(tokens)[-1]        # scores for the next token only
        probs = softmax(logits)           # a distribution over the vocabulary
        next_token = sample(probs)        # temperature, top-p, greedy: your choice
        tokens.append(next_token)         # feed the output back in as input
        if next_token == EOS:
            break
    return tokens

That loop is the whole of "AI writing an essay." The model never plans the paragraph. It picks a token, looks at what it just wrote, and picks again. Everything that looks like reasoning, style, or intent is the accumulated shape of that distribution, learned from data. The sampling step is a separate dial you control at inference time, which is why the same weights can be near-deterministic for extraction and loose for brainstorming.

The architecture computing that distribution is almost always a decoder-only transformer, but the architecture is an implementation detail of the function. The training pipeline is what makes the function useful.

Stage one: pretraining buys knowledge, and nothing else

Pretraining is next-token prediction over a very large corpus of text and code, on the order of trillions of tokens. The loss is plain cross-entropy: at every position, how surprised was the model by the token that actually came next. There is no human in the loop, no labels, no notion of a good answer. The supervision is the text itself, which is why this scales to the whole internet.

What you get out is a base model. It has absorbed grammar, facts, code idioms, translation, arithmetic patterns, and a great deal of world structure, because predicting the next token well eventually requires modeling the thing the text is about. This is where capability lives. It is also expensive: a frontier pretraining run means thousands of accelerators for weeks or months, and a budget in the millions of dollars. Scaling laws exist to tell you how to spend it.

What you do not get is an assistant. Ask a base model "What is the capital of France?" and a perfectly reasonable continuation is another question, because in the training corpus, a question is often followed by more questions:

Prompt:  What is the capital of France?
Base:    What is the capital of Germany? What is the largest city in Spain? ...

The base model is not being unhelpful. It is doing exactly what it was trained to do: continue the document. It has never once been shown what "responding" means.

Stage two and three: post-training installs the behavior

Supervised fine-tuning (SFT) is the same next-token loss, run on a small, curated set of (prompt, ideal response) pairs written or vetted by humans. Tens of thousands of examples is a normal order of magnitude. The model is not learning France's capital here, it already knew that. It is learning the shape of the interaction: when a turn ends, that a question deserves an answer, that a request for a bulleted list produces a bulleted list. This is where the base model stops continuing and starts responding.

Preference alignment is the last stage. SFT teaches one good response per prompt; it cannot teach that response A is better than response B when both are acceptable. So you collect comparisons (humans pick the better of two model outputs) and optimize the model toward the preferred one, either through a reward model plus RL (RLHF) or directly on the pairs (DPO). This is what installs tone, refusal behavior, hedging, formatting instincts, and the reluctance to make things up.

The compute asymmetry is the part people miss. Post-training runs on the order of one percent of pretraining compute: the canonical InstructGPT figure puts the whole RLHF stage at roughly 1.6% of the pretraining budget, and modern reasoning models with heavy RL post-training are pushing that share up. A small slice of extra training reorganizes how a model behaves without meaningfully adding to what it knows. InstructGPT is also the classic demonstration of the effect: a 1.3B aligned model was preferred by human raters over the 175B base model. The small model did not know more. It behaved better.

The load-bearing idea

Capability comes from pretraining. Behavior comes from post-training. Hold that and you can triage almost any production failure on the spot:

Most tickets are behavior tickets, which is why most of them are fixable with prompting, decoding constraints, or a light fine-tune, and why so few of them are fixable by training a bigger model. The one category that genuinely resists post-training is missing capability: if the base model cannot do the reasoning, no amount of alignment data will install it.

Why interviewers probe this

This is the load-bearing screen. Every deeper topic (attention, KV cache, RLHF, RAG) assembles into this picture, and a candidate who can state the picture cleanly is one who understands the parts rather than reciting them. The strong-answer move is to draw the boundary out loud: pretraining gives knowledge and fluency and produces a text continuer, post-training gives behavior and produces an assistant, and the two answer different bug reports. The weak answer describes an LLM as "a neural network trained on lots of text to understand language," which explains nothing and leaves the candidate unable to say why a base model does not answer questions.

The follow-up held in reserve is usually diagnostic: "our model keeps hallucinating our customers' order IDs, should we fine-tune?" The right answer is no, that is a knowledge problem and fine-tuning is the wrong tool for volatile facts, retrieval is. Candidates who reach for fine-tuning by reflex have not internalized which stage owns what.

Common misconceptions

• "The model plans its answer, then writes it." It samples one token at a time, conditioned on everything so far, including its own previous tokens. Chain-of-thought works precisely because the intermediate tokens become inputs the model can condition on.
• "Fine-tuning teaches the model new facts." It mostly teaches behavior and format. Facts injected by fine-tuning are learned unreliably, go stale, and cannot be cited. Put facts in context.
• "ChatGPT is what you get from pretraining." Pretraining gives a base model that continues documents. Every trait you associate with a chat assistant was installed after pretraining.
• "RLHF makes the model smarter." It makes the model behave better. Capability is set at pretraining; alignment reorganizes access to it, and heavy alignment can even cost a little raw capability, the so-called alignment tax.
• "It is a search engine over its training data." Nothing is retrieved. The weights are a lossy compression of the corpus, which is exactly why the model can be fluent and confidently wrong at the same time.

Key takeaways

• An LLM is a function from a token sequence to a distribution over the next token; generation is that function run in a loop on its own output.
• Pretraining is unsupervised next-token prediction on trillions of tokens, and it buys knowledge and fluency while producing a model that continues text rather than answers.
• SFT on tens of thousands of demonstrations teaches it to respond; preference alignment (RLHF or DPO) teaches it which response a human prefers.
• Post-training costs on the order of one percent of pretraining compute (InstructGPT's RLHF stage was about 1.6% of its pretraining budget) yet accounts for nearly everything users perceive as the model's personality.
• Capability comes from pretraining, behavior comes from post-training, and most production problems are behavior problems.