Skip to content
Antonio Kodheli

How to Build a RAG Chatbot on Your Own Data (2026)

Build a RAG chatbot step by step: chunking, embeddings, pgvector, hybrid search, reranking and cited answers with Claude, plus real monthly costs.

Antonio Kodheli··11 min read

A RAG chatbot answers questions from your own documents. When someone asks a question, it searches your content for the most relevant passages, gives those passages to a language model, and the model writes an answer based on them, ideally with citations. RAG stands for retrieval-augmented generation. It is the standard way to make a chatbot know your business without retraining a model, and it is the most requested AI feature I build for clients.

The top results for this search are mostly a YouTube video, a template repo and short explainers. This is the full build, the way I'd do it for a business that needs answers it can trust: every step, the code, what it costs to run each month, and the mistakes that make RAG chatbots give wrong answers.

The RAG ladder: from demo to production

Most RAG tutorials stop at step 2. The answer quality people expect from a business chatbot starts at step 4.

StepWhat you addWhat it fixesLabel
1Whole documents in the promptNothing to buildBest for: small, stable knowledge bases
2Chunks + vector searchScales to large contentBest for: demos and prototypes
3Hybrid search (vectors + keywords)Misses on exact terms: SKUs, error codes, namesBest for: product and support content
4Contextual chunks + rerankingChunks that lose meaning out of context; noisy top resultsBest for: production business chatbots
5Citations, permissions, evalsTrust, data leaks, silent quality dropsBest for: anything customers or staff rely on

Anthropic measured steps 3 and 4 on real retrieval tasks: adding context to each chunk plus keyword (BM25) search cut retrieval failures by 49%, and adding a reranker cut them by 67% (Anthropic, contextual retrieval). That is the difference between a chatbot that's right most of the time and one people trust.

How a RAG chatbot works

  INDEXING (runs when content changes)
  docs ──▶ clean ──▶ chunk ──▶ add context ──▶ embed ──▶ Postgres + pgvector
                                                          (+ keyword index)

  ANSWERING (runs on every question)
  question ──▶ hybrid search ──▶ top 30 chunks ──▶ rerank ──▶ top 6
           ──▶ Claude: "answer only from these sources, cite them"
           ──▶ answer with citations ──▶ log for evaluation

Step 1: Collect and clean your content

List every source the chatbot should answer from: help center, PDFs, policies, product catalog, past tickets. For each, decide who may see it. Then clean it: strip navigation and boilerplate, keep headings, convert tables to text the model can read, and record the source URL and last-updated date for every document.

Label: the step that decides quality. Bad or outdated source content is the most common reason a RAG chatbot is wrong. No retrieval trick fixes a wrong policy page.

Step 2: Chunk it

Split documents into passages small enough to retrieve precisely but large enough to make sense alone. Start with 300 to 800 tokens per chunk, split on headings and paragraphs rather than a fixed character count, with a small overlap between neighbours.

Then add context. A chunk that says "Refunds take 5 to 7 days" is ambiguous on its own. Prefix each chunk with a line or two about where it comes from ("From the Returns policy, section on card refunds:"). Generate that line once per chunk with a cheap model at indexing time. This is the "contextual" part of the 49% result above.

Step 3: Embed and store

An embedding model turns each chunk into a vector, a list of numbers that captures its meaning, so similar meanings end up close together. Claude doesn't make embeddings; Anthropic points developers to Voyage AI, and OpenAI and Google offer embedding models too. Pick one and use it for both indexing and questions.

Store chunks in Postgres with the pgvector extension. You likely already run Postgres, and one database for content, vectors and permissions is easier to secure than three services. I use the same setup for long-term memory in Pair.

create extension if not exists vector;

create table chunks (
  id          bigserial primary key,
  doc_url     text not null,
  doc_title   text not null,
  audience    text not null default 'public',   -- who may see it
  updated_at  date not null,
  content     text not null,                    -- context line + chunk text
  embedding   vector(1024) not null,            -- match your embedding model's size
  tsv         tsvector generated always as (to_tsvector('english', content)) stored
);

create index on chunks using hnsw (embedding vector_cosine_ops);
create index on chunks using gin (tsv);

Vector search finds passages with the same meaning. Keyword search finds exact terms that vectors blur: product codes, error messages, people's names. Run both and merge the results with reciprocal rank fusion:

with semantic as (
  select id, row_number() over (order by embedding <=> $1) as r
  from chunks where audience = any($3)
  order by embedding <=> $1 limit 30
),
keyword as (
  select id, row_number() over (order by ts_rank(tsv, q) desc) as r
  from chunks, websearch_to_tsquery('english', $2) q
  where tsv @@ q and audience = any($3)
  order by ts_rank(tsv, q) desc limit 30
)
select c.id, c.doc_title, c.doc_url, c.content,
       coalesce(1.0 / (60 + s.r), 0) + coalesce(1.0 / (60 + k.r), 0) as score
from chunks c
left join semantic s on s.id = c.id
left join keyword  k on k.id = c.id
where s.id is not null or k.id is not null
order by score desc
limit 30;

$1 is the question's embedding, $2 the question text, $3 the audiences this user may see. Filter permissions in the query, never in the prompt. A model told "don't mention internal documents" can still leak them; a model that never receives them can't.

Step 5: Rerank

Send the top 30 candidates to a reranking model, which scores each one against the question far more precisely than vector distance, and keep the best 5 to 8. Rerankers are cheap and the jump from 49% to 67% fewer failures in Anthropic's test came from this step.

Step 6: Generate a cited answer with Claude

Give the model the question and the reranked passages, numbered, and strict instructions:

import Anthropic from "@anthropic-ai/sdk";
const anthropic = new Anthropic();

const sources = top
  .map((c, i) => `<source id="${i + 1}" title="${c.doc_title}" url="${c.doc_url}">\n${c.content}\n</source>`)
  .join("\n");

const msg = await anthropic.messages.create({
  model: "claude-haiku-5-5",
  max_tokens: 800,
  system:
    "You answer customer questions for Acme using ONLY the sources provided. " +
    "Cite sources inline like [1]. If the sources don't contain the answer, say you don't know " +
    "and offer to connect the customer with a person. Never guess prices, dates or policies.",
  messages: [{ role: "user", content: `${sources}\n\nQuestion: ${question}` }],
});

Three rules do most of the work: answer only from the sources, cite them, and say "I don't know" instead of guessing. Show the citations as links in the chat UI so people can check. Claude's API also has a built-in citations feature that returns the exact passages it used, which is worth turning on when answers must be auditable.

Step 7: Evaluate before and after launch

Collect 50 real questions with known correct answers, including ones the content doesn't cover. Score two things on every change:

  • Retrieval: did the right passage make the top 6?
  • Answer: was it correct, cited, and did it refuse when it should?

Log every live question, the passages retrieved and the answer. Weekly, read the ones users rated badly or that ended in "I don't know". They tell you which content is missing.

What a RAG chatbot costs to run

Using Claude API prices checked on November 20, 2026, and this assumption: each answer reads about 6,000 input tokens (instructions plus 6 passages) and writes 400 output tokens.

ModelPer answer10,000 questions a month
Claude Haiku 5.5 ($0.10 / $0.50 per million tokens)about $0.0008about $8
Claude Sonnet 5.5 ($2 / $10 per million tokens)about $0.016about $160

Add embeddings (a one-time cost to index, then small updates), a reranker, and hosting. For most small and mid-size businesses, the model bill is the small part; building it well and keeping the content current is the real cost. More in how much an AI agent costs and Claude API pricing.

Tip: start on Haiku and move individual question types to Sonnet only where your evaluation shows Haiku getting them wrong.

RAG vs fine-tuning

RAGFine-tuning
Teaches the modelFacts it can look upStyle, format, narrow behaviour
Updating knowledgeRe-index changed documents, minutesRetrain, days
CitationsYes, every answer can point to a sourceNo
Hallucination controlStrong: "answer only from sources"Weak: facts are blended into weights
Access controlPer document, per userNone
LabelBest for: knowledge that changes, answers you must verifyBest for: consistent tone or output format at high volume

For a business chatbot, the answer is almost always RAG. Fine-tune only after RAG and good instructions can't get the style right.

Common mistakes

  1. Chunking by character count. Splits sentences and tables mid-thought. Split on structure.
  2. Vector search only. Misses exact codes and names. Use hybrid.
  3. Permissions in the prompt. Filter in the database query.
  4. No "I don't know". The model will fill gaps with confident guesses.
  5. No evaluation set. You won't notice when a content update breaks answers.
  6. Stale content. Re-index on every change and show "last updated" on cited sources.

Keep going

Frequently asked questions

What is a RAG chatbot? A chatbot that answers from your own content. It retrieves the most relevant passages from your documents for each question and has a language model write an answer based only on them, usually with citations.

How do I build a RAG chatbot? Collect and clean your content, split it into chunks, embed the chunks into a vector database such as Postgres with pgvector, retrieve with hybrid search, rerank the results, then have a model like Claude answer from the top passages with citations. Test it against real questions before launch.

How much does a RAG chatbot cost to run? Model usage can be very low. At about 6,000 input and 400 output tokens per answer, 10,000 questions a month cost about $8 on Claude Haiku 5.5 or about $160 on Claude Sonnet 5.5, plus embeddings, reranking and hosting.

What is the difference between RAG and fine-tuning? RAG gives the model facts to look up at question time and can cite them; fine-tuning changes the model's weights to teach style or behaviour. Use RAG for knowledge that changes, fine-tuning for tone or format.

Do I need a vector database for RAG? Not always. If your content fits comfortably in the model's context window, put it all in the prompt and use prompt caching. Use a vector database once the content is too large or changes often.

How do I stop a RAG chatbot from making things up? Instruct it to answer only from the retrieved sources and to say when it doesn't know, show citations, improve retrieval with hybrid search and reranking, and run an evaluation set on every change.

I build RAG chatbots and AI assistants that answer from your documents, help center or product data, with citations and access control. Tell me what it should answer and you'll get a fixed price and a monthly running-cost estimate.

Get a RAG chatbot built on your data

Written by Antonio Kodheli

Full-stack web developer in Boston. I build modern web apps with TypeScript, Next.js, and the Claude API, and write about it here.