Introduction

In AI Internals - Essentials we split the world into a model stack and an app stack, and left the model side as a black box: messages in, completion out.

A large language model (LLM) is a next-token predictor with a fixed vocabulary, a fixed context budget, and a sampling step that turns a probability distribution into the string you see in the response. Once those pieces are clear, API parameters stop being folklore: temperature, max_tokens, and "context too long" errors point at concrete stages in the pipeline.

This article stays on the model stack: tokens, training versus inference, the transformer decode loop, context windows and the key-value (KV) cache, and sampling. Prompts as a control surface come next in Prompts & Context.

Tokens are the real alphabet

You write strings. The model never sees characters or words as first-class units. A tokenizer maps text to a sequence of integer token IDs from a fixed vocabulary (often tens or hundreds of thousands of entries). The reverse map turns IDs back into text for the completion.

Modern chat models usually use a subword scheme — Byte Pair Encoding (BPE) or a close relative (SentencePiece, tiktoken-style BPE). Frequent strings become single tokens (the, ing, https). Rare or weird strings split into smaller pieces, sometimes down to bytes.

That has practical consequences you hit in production:

• Token count is not character count and not word count. English prose often lands around ~0.7–1.3 tokens per word; code, JSON, and non-English text can look very different.
• Tiny text edits can change tokenization a lot. "hello" and " hello" (leading space) are often different token sequences. So are "JSON" vs "json".
• Pricing, rate limits, and context windows are almost always in tokens, not characters. Guessing from len(prompt) will eventually bite you.

You do not need to reimplement BPE to build apps. You do need to measure tokens for anything that touches cost or context. Provider responses often return usage.prompt_tokens and usage.completion_tokens. Local counting tools (tiktoken for OpenAI-compatible vocabs, Hugging Face tokenizers for open models) matter when you are packing retrieval-augmented generation (RAG) chunks or trimming history before the request fails.

Here is the same completion call as in Essentials, but reading the usage the provider already computed:

usage is post-hoc. If you need to know whether a 40k-token RAG pack will fit before you pay for the call, count locally with the same tokenizer family the model uses. Mismatched tokenizers give you a wrong budget.

Special tokens also matter at the edges: chat templates wrap your messages with role markers the model was trained on (<|im_start|>, etc.). Chat Completions APIs usually apply that template for you. Raw completion endpoints and local servers often make you own it — wrong template, worse answers, same weights.

Next-token prediction

Under the hood, a decoder-only LLM is trained to answer one question repeatedly: given tokens t1..tn, what is the distribution over the next token t(n+1)?

Training minimizes how surprised the model is by the real next token in large corpora (and later, by preferred answers in fine-tuning). At inference time the weights are frozen. The model still emits a distribution over the vocabulary; software samples (or argmaxes) one ID, appends it, and repeats until a stop condition.

That single loop explains a lot of product behavior:

• The model does not "look up an answer" and then type it. It grows the answer one token at a time, conditioning on everything already in the context — including its own earlier tokens.
• Early mistakes condition later ones. A wrong tool name in token 3 can poison the rest of the call.
• Latency is often dominated by how many tokens you generate (decode), not only by how big the prompt is (prefill) — more on that below.

People say "the model knows X." Operationally it means: after training, the weights implement a conditional distribution that puts high probability on certain continuations when the context looks a certain way. There is no separate knowledge base inside the API call unless you put documents or tool results into the messages (later articles).

Training versus inference

These are different jobs that share architecture and vocabulary.

Pretraining runs on huge text (and code) corpora. The objective is language modeling: predict the next token. This is where most of the raw capability and "world knowledge" shape comes from. It is also where the model learns syntax, common facts that appeared often enough, and a lot of internet junk.

Post-training adapts that base model toward useful assistants:

• Supervised fine-tuning (SFT) — show input → desired output pairs (chat format, tool-use traces, refusal styles).
• Preference / reinforcement learning (RL)-style stages — Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), Reinforcement Learning from AI Feedback (RLAIF), and cousins — push the distribution toward answers humans or judges rank higher, and away from disallowed or low-quality ones.

You almost never run these stages in an app. You pick a checkpoint a provider or open-weight project already shipped. Your job at inference is to steer that fixed distribution with context and decoding settings.

Fine-tuning still exists as a product feature, but it is optional and expensive relative to prompt and retrieval work. Most systems in this series assume a frozen hosted model.

One more distinction that shows up in serving stacks: prefill versus decode.

• Prefill — process the prompt tokens, usually in parallel across positions, and build the initial KV cache (cached key and value tensors attention will reuse).
• Decode — generate new tokens one by one (or in small speculative chunks). Each step reuses the cache and appends one position.

A long system prompt plus a big RAG dump makes prefill heavy. A long assistant answer makes decode heavy. Streaming UIs feel "alive" during decode because tokens arrive incrementally; the silent pause at the start is often prefill.

Inside one forward pass (enough to be dangerous)

You do not need to derive attention from scratch to ship features. You do need a picture of what burns compute and what the context window is paying for.

A typical decoder block stacks:

1. Causal self-attention — each position builds a query and attends to keys/values of earlier positions (and itself), not future ones. That causal mask is why the model can be trained with teacher forcing and still run left-to-right at inference.
2. Feed-forward / multilayer perceptron (MLP) — per-position nonlinear transform; a large fraction of the parameter count lives here.
3. Residuals and normalization — keep deep stacks trainable.

Attention is where context becomes real. Information from token 2 can influence the distribution at token 500 if both still sit in the window. It is also where cost grows with sequence length: naive attention is O(n²) in the number of tokens for the pairwise scores (serving stacks use kernels, paging, and other tricks; the scaling pressure remains).

Embeddings at the input are just a table: token ID → vector. Positional information is mixed in (Rotary Position Embedding (RoPE) and friends in modern models) so order is not lost. The final layer maps the last hidden state to logits — one raw score per vocabulary entry — and that vector is what sampling consumes.

Multimodal models add encoders that turn images or audio into token-like embeddings in the same residual stream. From the app's point of view they still consume context budget and still produce next-token distributions; the tokenizer story just gets a visual or audio sibling.

Context windows and the KV cache

The context window is the maximum number of tokens the serving stack will hold for one generation — prompt plus completion, with provider-specific accounting for special tokens and sometimes for tool payloads.

When you exceed it, the API errors or the server truncates. There is no soft infinite memory inside a single call. Multi-turn chat feels like memory only because your app (or the provider's session layer) resends prior messages each turn. Every resend pays tokens again.

Why a hard limit exists:

• Attention and cache storage grow with sequence length.
• Quality often degrades when you stuff the window with weakly relevant text ("lost in the middle" effects show up in evals).
• Product tiers are priced and capacity-planned around maximum context.

The KV cache is the inference-time structure that makes decode affordable. For each layer and each past token position, the server stores keys and values so it does not recompute the whole prompt on every new token. Cache size scales with layers × heads × dimensions × sequence length × precision. Long contexts are a memory problem as much as a floating-point operations (FLOPs) problem — that is why batch size per GPU drops as context grows, and why "128k context" models still bill you for using it.

Practical rules that fall out of this:

• Put stable instructions once; do not duplicate them in every retrieved chunk.
• Summarize or drop old turns instead of appending forever.
• Retrieval exists so you do not paste entire corpora into the window — that is the bridge to Embeddings & Retrieval and RAG.
• max_tokens (or max_completion_tokens) caps output length; it does not enlarge the window. Prompt tokens plus max output must still fit.

Finish reasons on the response tell you how the loop stopped: stop (natural / stop sequence), length (hit the output cap), content_filter, tool calls, and so on. length usually means you truncated mid-thought — often worse than asking for a shorter answer up front.

Logits to text: sampling

After the forward pass you have logits z over the vocabulary. Greedy decoding picks the single highest logit. Most chat UIs do not do that by default — they sample, which is why the same prompt can yield different answers.

Common controls (names vary by API; the math is shared):

Temperature scales logits before softmax. Softmax turns scaled logits into probabilities p_i ∝ exp(z_i / T). Lower temperature sharpens the distribution; higher flattens it.

• T → 0 approaches greedy (numerically you just argmax).
• T = 1 leaves logits as trained.
• T > 1 spends more probability on the long tail — creative, unstable, sometimes nonsense.

Top-k keeps only the k highest logits, renorms, then samples.

Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability is at least p, renorms, then samples. It adapts to peaked versus flat distributions better than a fixed k.

Stop sequences end decode when the model emits a chosen string (or token sequence). Chat APIs also stop on role boundaries their templates define.

Presence / frequency penalties (where exposed) adjust logits to discourage repetition. They are blunt instruments; better prompts and lower temperature often fix repetition with less collateral damage.

Compare two temperatures on a short creative prompt and log the text. Keep everything else fixed (top_p, model, seed if the API supports it):

For extraction, classification, and most tool-calling control loops, low temperature (often 0 or close) is the default that wastes less time. Save high temperature for ideation where diversity is the point and you will filter or rank outputs yourself.

Determinism is weaker than people expect. temperature: 0 is usually greedy-ish, but provider stacks can still vary across replicas, batching, and sparse Mixture of Experts (MoE) routing. If you need reproducible evals, pin model version, record parameters, and treat bitwise equality as a bonus — not a guarantee.

Mapping API knobs to the pipeline

A Chat Completions-style request is a thin wrapper over the stages above:

Structured output and JSON mode are not a second brain. They constrain or bias which tokens are legal while sampling so the continuation is more likely to parse. The model can still emit confident nonsense inside a valid JSON shape — which is why grounding and evals matter later.

Streaming (stream: true) does not change the math. It changes delivery: the server flushes tokens as decode produces them. Your client still concatenates to the same final string; you just get TTFT (time to first token) and a progressive UI.

Failure modes that are really model-stack issues

Before you rewrite the prompt for the fifth time, check whether the failure is mechanical:

• Context overflow — prompt + reserved output exceeds the window. Shrink history, chunks, or max_tokens.
• Truncated JSON / mid-sentence cut — finish_reason is length. Raise output budget or demand shorter structure.
• Wild variance run to run — temperature / top-p too high for a control task.
• Repetition loops — decoding pathology; lower temperature, penalties, or stop sequences; sometimes a shorter context helps.
• Tokenizer surprises — cost or truncation math wrong because you counted characters or used the wrong vocab.
• Template mismatch — local model with a chat template that does not match how it was trained.

App-level failures (bad retrieval, missing tools, weak evals) dominate once the call actually runs. Those get their own articles. The point here is not to blame the model for everything — it is to recognize when the bottleneck is tokens, window, or sampling.

Conclusion

An LLM call is a tokenize → prefill → repeated sample-and-append → detokenize loop over a frozen next-token distribution. Context size is a hard budget backed by attention and KV-cache memory. Temperature and friends only reshape the distribution you sample from; they do not add knowledge.

Next up is Prompts & Context — messages, system prompts, and structured output as the control surface on top of this machinery.