Introduction

In AI Internals - Embeddings & Retrieval we stopped at search: embed passages, index them, fetch neighbors, optionally fuse and rerank. The generative model never entered the picture.

RAG (retrieval-augmented generation) is the wiring: take those neighbors, pack them into the message contract from Prompts & Context, and make the completion depend on that evidence. Chunking decides what units exist to retrieve. Grounding decides whether the model is allowed to invent past them.

Most demos treat RAG as "paste top-k into the prompt." Production pain lives in how you split documents, how you budget tokens, and what you do when retrieval returns nothing useful.

RAG is a pipeline, not a feature flag

Retrieval alone answers "which passages look close to the query?" Generation alone answers "what string is likely next?" RAG answers a third question: "given these passages as temporary knowledge, produce an answer the app can trust."

Three properties fall out of that loop:

• Freshness comes from your index, not from pretraining. Update the doc store and answers can change without a fine-tune.
• Attribution is possible only if you keep IDs with the text you inject and ask for them back (or attach them yourself).
• Failure is composable. A wrong chunk, a truncated chunk, a missing citation rule, or a model that ignores the evidence all look like "RAG is broken" from the outside.

Fine-tuning still has a job — style, tool schemas, domain jargon that should live in weights — but it is a poor substitute for a knowledge base that changes weekly. RAG keeps facts outside the frozen model.

Chunking: the unit of truth

In the previous article a "passage" was already a string with an ID. Here we care how that string was cut from a source document.

Chunking is irreversible for the index: if the right sentence never coexists in the same unit with the header that gives it meaning, dense search cannot invent that pairing later. Rerankers and better prompts will not fix a bad cut.

Size is a trade-off, not a constant

Small chunks (roughly 100–300 tokens) improve precision: the neighbor is more likely to be about one thing. They also lose surrounding definitions, table headers, and "this section applies to annual plans only" caveats that lived two paragraphs up.

Large chunks (roughly 800–1500 tokens) keep more local context and survive better when the answer spans a few sentences. They waste context budget when only one sentence mattered, and they dilute embedding signal so near-miss neighbors climb the ranking.

There is no universal number. Calibrate on your corpus and a labeled query set: measure retrieval recall@k and answer faithfulness, not just "chunks feel about right."

Boundaries beat character counts

Sliding windows of N characters with overlap are a reasonable baseline and a common source of garbage. Prefer structural cuts when the source has them:

• Markdown / HTML headings as hard breaks
• Paragraph or sentence boundaries inside a section
• Code at function or type boundaries, not mid-line
• Tables kept intact or summarized as a row-oriented unit with the column legend attached

Overlap (for example 10–20% of chunk size) helps when a sentence straddles a cut. It also duplicates text in the index and can make the same fact appear twice in the packed context — budget for that.

Parent–child and metadata

A pattern that shows up often in production knowledge bases:

1. Child chunks — small retrieval units for precise matching.
2. Parent documents — larger blocks (section or page) stored by ID.
3. At query time, retrieve children, then expand to the parent (or to neighboring children) before packing.

You pay a bit more storage and join logic. You avoid answering from a floating sentence that only made sense under its ## Refunds heading.

Metadata on every unit matters as much as the text: doc_id, section_title, url, updated_at, tenant_id, product version. Filters from the retrieval article apply here; version skew ("docs for v2, product is on v3") is a RAG failure even when similarity scores look fine.

source: docs/billing.md
section: ## Annual plan refunds
chunk_id: billing.md#annual-plan-refunds:2
text: "Annual plans can be refunded within 14 days of purchase..."
meta: { product: "billing", version: "2026-07", url: "..." }

If you cannot point from a chunk back to a URL the user can open, citations in the answer are theater.

From hits to a prompt

Retrieval returns candidates. The prompt assembler decides what the model actually sees.

Reuse the message contract from Prompts & Context:

System:
- Role and hard constraints
- Grounding rule: answer only from Evidence; say you lack evidence when it is missing
- Citation rule: if you claim a fact, attach chunk IDs

User:
- Objective
- Evidence: numbered snippets with IDs
- Question

A concrete packing order that usually works:

1. Reserve tokens for system instructions and the user question (count with the same tokenizer family as the chat model).
2. Fill the remaining budget with reranked snippets, highest first.
3. Drop the rest. Do not silently truncate mid-snippet if you can drop whole units instead — half a table is worse than omitting it.
4. Deduplicate near-identical overlap chunks before packing.

Latency and cost scale with packed tokens (prefill) plus completion length. Retrieving 50 and packing 8 is normal; packing all 50 "just in case" burns money and often hurts quality when noise dilutes the evidence.

Grounding: making evidence binding

Injecting text is not the same as grounding. Models will still blend pretrained priors with your snippets unless the instructions and evals push the other way.

Practical levers:

• Explicit refuse path. "If Evidence does not contain the answer, say you do not know and ask for a doc link — do not guess." Without that, empty or wrong retrieval still produces fluent fiction.
• Separate Evidence from Question visually. Labels, fences, or XML-ish tags reduce the chance the model treats your policy text as optional flavor.
• Citations as a product requirement. Ask for [chunk_id] markers, then verify them in code against the IDs you actually sent. Unverified citation strings are easy to fake.
• Low temperature for factual answer modes. Sampling creativity fights grounding.
• Structured output when the app needs answer, citations[], confidence fields instead of free prose.

Grounding is also a product policy. Support bots may paraphrase. Legal or medical surfaces may need quote-level fidelity and human review. The pipeline is the same; the contract in system changes.

A minimal RAG request shape

Below is the shape of packing retrieved rows into a chat call. Retrieval is assumed done (hybrid + rerank from the previous article); this is the generation-side half.

package main

import (
  "bytes"
  "encoding/json"
  "fmt"
  "io"
  "net/http"
  "os"
  "strings"
)

type Chunk struct {
  ID   string
  Text string
}

func packEvidence(chunks []Chunk, maxChars int) string {
  var b strings.Builder
  used := 0
  for _, c := range chunks {
    block := fmt.Sprintf("[%s]\n%s\n\n", c.ID, c.Text)
    if used+len(block) > maxChars {
      break
    }
    b.WriteString(block)
    used += len(block)
  }
  return b.String()
}

func main() {
  chunks := []Chunk{
    {
      ID:   "billing.md#refunds:1",
      Text: "Annual plans can be refunded within 14 days of purchase.",
    },
    {
      ID:   "billing.md#api-keys:1",
      Text: "API keys are rotated from the developer settings page.",
    },
  }

  evidence := packEvidence(chunks, 2000)
  question := "Can I get a refund on a yearly plan bought last week?"

  body, _ := json.Marshal(map[string]any{
    "model": "gpt-4.1-mini",
    "temperature": 0.2,
    "messages": []map[string]string{
      {
        "role": "system",
        "content": "Answer using only Evidence. If Evidence is insufficient, say you lack evidence. Cite chunk IDs in square brackets.",
      },
      {
        "role": "user",
        "content": "Evidence:\n" + evidence + "Question: " + question,
      },
    },
  })

  req, _ := http.NewRequest(
    "POST",
    "https://api.openai.com/v1/chat/completions",
    bytes.NewReader(body),
  )
  req.Header.Set("Authorization", "Bearer "+os.Getenv("OPENAI_API_KEY"))
  req.Header.Set("Content-Type", "application/json")

  res, err := http.DefaultClient.Do(req)
  if err != nil {
    panic(err)
  }
  defer res.Body.Close()
  raw, _ := io.ReadAll(res.Body)
  fmt.Println(string(raw))
}

Char budgets are a blunt stand-in for token budgets. In a real service, count tokens for the model you call and leave headroom for the completion (max_tokens).

Query-side tricks that belong to RAG

Pure retrieval embeds the user string as-is. RAG systems often reshape the query before search:

• Rewrite — turn "that thing from yesterday" plus chat history into a standalone search string.
• Multi-query — fan out paraphrases, union the hit lists, then rerank once.
• HyDE-style — have the model draft a hypothetical answer passage, embed that, and search (helps some corpora; can also retrieve confident nonsense if the draft is wrong).

These steps cost extra LLM or embedding calls. Add them when metrics show the raw user utterance is a bad search key — short, anaphoric, or jargon-heavy — not because a blog post listed them.

Failure modes that only show up after wiring

Part 4 covered empty indexes and wrong neighbors. Once generation is attached, new bugs appear:

• Right retrieval, wrong answer. Evidence was packed; the model ignored it or overrode it with pretrained fluff. Tighten grounding rules, lower temperature, verify citations in code.
• Answer cites IDs you never sent. Treat as a failed response. Do not show fake footnotes in the UI.
• Chunk boundary amputated the rule. Retrieval score looked high; the sentence with the 14-day limit lived in the next child chunk. Expand to parent or widen overlap.
• Stale parent, fresh child (or the reverse) after partial reindex. Version the corpus and rebuild consistently.
• Context stuffing. Too many middling chunks; the model hedges or mixes conflicting policies. Pack fewer, better units after rerank.
• Silent policy conflict. Two retrieved snippets disagree; the model picks one without flagging. Instruct it to surface conflicts, or resolve at retrieval with metadata (prefer newest updated_at).
• "I don't know" never triggers. Refuse path missing or contradicted elsewhere in the system prompt. Users then get invented SLAs.

Debug habit: log query, rewritten query, retrieved IDs with scores, packed IDs, and the final completion. If you only store the user-visible answer, you cannot tell retrieval bugs from grounding bugs.

Comparison: how far to take the pipeline

You do not need a graph framework to start. You need measurable chunking, a packing budget, and grounding rules the app enforces — not only the model pinky-promises.

Conclusion

RAG is retrieval plus disciplined context construction: cut documents into units that keep meaning, fetch them with the search stack from the previous article, pack under a real token budget, and bind the completion to that evidence with refuse and citation rules you verify in code.

Next up is Tools & Agents — when the model must take actions (not only read snippets), how tool calls fit the same message loop, and when an agent loop is the wrong shape.