How to Build a RAG Chatbot on Your Own Data (2026)
Build a RAG chatbot step by step: chunking, embeddings, pgvector, hybrid search, reranking and cited answers with Claude, plus real monthly costs.
A RAG chatbot answers questions from your own documents. When someone asks a question, it searches your content for the most relevant passages, gives those passages to a language model, and the model writes an answer based on them, ideally with citations. RAG stands for retrieval-augmented generation. It is the standard way to make a chatbot know your business without retraining a model, and it is the most requested AI feature I build for clients.
The top results for this search are mostly a YouTube video, a template repo and short explainers. This is the full build, the way I'd do it for a business that needs answers it can trust: every step, the code, what it costs to run each month, and the mistakes that make RAG chatbots give wrong answers.
The RAG ladder: from demo to production
Most RAG tutorials stop at step 2. The answer quality people expect from a business chatbot starts at step 4.
| Step | What you add | What it fixes | Label |
|---|---|---|---|
| 1 | Whole documents in the prompt | Nothing to build | Best for: small, stable knowledge bases |
| 2 | Chunks + vector search | Scales to large content | Best for: demos and prototypes |
| 3 | Hybrid search (vectors + keywords) | Misses on exact terms: SKUs, error codes, names | Best for: product and support content |
| 4 | Contextual chunks + reranking | Chunks that lose meaning out of context; noisy top results | Best for: production business chatbots |
| 5 | Citations, permissions, evals | Trust, data leaks, silent quality drops | Best for: anything customers or staff rely on |
Anthropic measured steps 3 and 4 on real retrieval tasks: adding context to each chunk plus keyword (BM25) search cut retrieval failures by 49%, and adding a reranker cut them by 67% (Anthropic, contextual retrieval). That is the difference between a chatbot that's right most of the time and one people trust.
How a RAG chatbot works
INDEXING (runs when content changes)
docs ──▶ clean ──▶ chunk ──▶ add context ──▶ embed ──▶ Postgres + pgvector
(+ keyword index)
ANSWERING (runs on every question)
question ──▶ hybrid search ──▶ top 30 chunks ──▶ rerank ──▶ top 6
──▶ Claude: "answer only from these sources, cite them"
──▶ answer with citations ──▶ log for evaluation
Step 1: Collect and clean your content
List every source the chatbot should answer from: help center, PDFs, policies, product catalog, past tickets. For each, decide who may see it. Then clean it: strip navigation and boilerplate, keep headings, convert tables to text the model can read, and record the source URL and last-updated date for every document.
Label: the step that decides quality. Bad or outdated source content is the most common reason a RAG chatbot is wrong. No retrieval trick fixes a wrong policy page.
Step 2: Chunk it
Split documents into passages small enough to retrieve precisely but large enough to make sense alone. Start with 300 to 800 tokens per chunk, split on headings and paragraphs rather than a fixed character count, with a small overlap between neighbours.
Then add context. A chunk that says "Refunds take 5 to 7 days" is ambiguous on its own. Prefix each chunk with a line or two about where it comes from ("From the Returns policy, section on card refunds:"). Generate that line once per chunk with a cheap model at indexing time. This is the "contextual" part of the 49% result above.
Step 3: Embed and store
An embedding model turns each chunk into a vector, a list of numbers that captures its meaning, so similar meanings end up close together. Claude doesn't make embeddings; Anthropic points developers to Voyage AI, and OpenAI and Google offer embedding models too. Pick one and use it for both indexing and questions.
Store chunks in Postgres with the pgvector extension. You likely already run Postgres, and one database for content, vectors and permissions is easier to secure than three services. I use the same setup for long-term memory in Pair.
create extension if not exists vector;
create table chunks (
id bigserial primary key,
doc_url text not null,
doc_title text not null,
audience text not null default 'public', -- who may see it
updated_at date not null,
content text not null, -- context line + chunk text
embedding vector(1024) not null, -- match your embedding model's size
tsv tsvector generated always as (to_tsvector('english', content)) stored
);
create index on chunks using hnsw (embedding vector_cosine_ops);
create index on chunks using gin (tsv);
Step 4: Retrieve with hybrid search
Vector search finds passages with the same meaning. Keyword search finds exact terms that vectors blur: product codes, error messages, people's names. Run both and merge the results with reciprocal rank fusion:
with semantic as (
select id, row_number() over (order by embedding <=> $1) as r
from chunks where audience = any($3)
order by embedding <=> $1 limit 30
),
keyword as (
select id, row_number() over (order by ts_rank(tsv, q) desc) as r
from chunks, websearch_to_tsquery('english', $2) q
where tsv @@ q and audience = any($3)
order by ts_rank(tsv, q) desc limit 30
)
select c.id, c.doc_title, c.doc_url, c.content,
coalesce(1.0 / (60 + s.r), 0) + coalesce(1.0 / (60 + k.r), 0) as score
from chunks c
left join semantic s on s.id = c.id
left join keyword k on k.id = c.id
where s.id is not null or k.id is not null
order by score desc
limit 30;
$1 is the question's embedding, $2 the question text, $3 the audiences this user may see. Filter permissions in the query, never in the prompt. A model told "don't mention internal documents" can still leak them; a model that never receives them can't.
Step 5: Rerank
Send the top 30 candidates to a reranking model, which scores each one against the question far more precisely than vector distance, and keep the best 5 to 8. Rerankers are cheap and the jump from 49% to 67% fewer failures in Anthropic's test came from this step.
Step 6: Generate a cited answer with Claude
Give the model the question and the reranked passages, numbered, and strict instructions:
import Anthropic from "@anthropic-ai/sdk";
const anthropic = new Anthropic();
const sources = top
.map((c, i) => `<source id="${i + 1}" title="${c.doc_title}" url="${c.doc_url}">\n${c.content}\n</source>`)
.join("\n");
const msg = await anthropic.messages.create({
model: "claude-haiku-5-5",
max_tokens: 800,
system:
"You answer customer questions for Acme using ONLY the sources provided. " +
"Cite sources inline like [1]. If the sources don't contain the answer, say you don't know " +
"and offer to connect the customer with a person. Never guess prices, dates or policies.",
messages: [{ role: "user", content: `${sources}\n\nQuestion: ${question}` }],
});
Three rules do most of the work: answer only from the sources, cite them, and say "I don't know" instead of guessing. Show the citations as links in the chat UI so people can check. Claude's API also has a built-in citations feature that returns the exact passages it used, which is worth turning on when answers must be auditable.
Step 7: Evaluate before and after launch
Collect 50 real questions with known correct answers, including ones the content doesn't cover. Score two things on every change:
- Retrieval: did the right passage make the top 6?
- Answer: was it correct, cited, and did it refuse when it should?
Log every live question, the passages retrieved and the answer. Weekly, read the ones users rated badly or that ended in "I don't know". They tell you which content is missing.
What a RAG chatbot costs to run
Using Claude API prices checked on November 20, 2026, and this assumption: each answer reads about 6,000 input tokens (instructions plus 6 passages) and writes 400 output tokens.
| Model | Per answer | 10,000 questions a month |
|---|---|---|
| Claude Haiku 5.5 ($0.10 / $0.50 per million tokens) | about $0.0008 | about $8 |
| Claude Sonnet 5.5 ($2 / $10 per million tokens) | about $0.016 | about $160 |
Add embeddings (a one-time cost to index, then small updates), a reranker, and hosting. For most small and mid-size businesses, the model bill is the small part; building it well and keeping the content current is the real cost. More in how much an AI agent costs and Claude API pricing.
Tip: start on Haiku and move individual question types to Sonnet only where your evaluation shows Haiku getting them wrong.
RAG vs fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| Teaches the model | Facts it can look up | Style, format, narrow behaviour |
| Updating knowledge | Re-index changed documents, minutes | Retrain, days |
| Citations | Yes, every answer can point to a source | No |
| Hallucination control | Strong: "answer only from sources" | Weak: facts are blended into weights |
| Access control | Per document, per user | None |
| Label | Best for: knowledge that changes, answers you must verify | Best for: consistent tone or output format at high volume |
For a business chatbot, the answer is almost always RAG. Fine-tune only after RAG and good instructions can't get the style right.
Common mistakes
- Chunking by character count. Splits sentences and tables mid-thought. Split on structure.
- Vector search only. Misses exact codes and names. Use hybrid.
- Permissions in the prompt. Filter in the database query.
- No "I don't know". The model will fill gaps with confident guesses.
- No evaluation set. You won't notice when a content update breaks answers.
- Stale content. Re-index on every change and show "last updated" on cited sources.
Keep going
- AI agent architecture: when a chatbot should become an agent with tools and memory.
- MCP vs API: letting the chatbot act on live data, not just documents.
- Claude Agent SDK and how to get a Claude API key.
- Building an AI website generator with Next.js and Claude: another end-to-end Claude build.
- Want it built? AI integration services, AI agent development or tell me what it should answer.
Frequently asked questions
What is a RAG chatbot? A chatbot that answers from your own content. It retrieves the most relevant passages from your documents for each question and has a language model write an answer based only on them, usually with citations.
How do I build a RAG chatbot? Collect and clean your content, split it into chunks, embed the chunks into a vector database such as Postgres with pgvector, retrieve with hybrid search, rerank the results, then have a model like Claude answer from the top passages with citations. Test it against real questions before launch.
How much does a RAG chatbot cost to run? Model usage can be very low. At about 6,000 input and 400 output tokens per answer, 10,000 questions a month cost about $8 on Claude Haiku 5.5 or about $160 on Claude Sonnet 5.5, plus embeddings, reranking and hosting.
What is the difference between RAG and fine-tuning? RAG gives the model facts to look up at question time and can cite them; fine-tuning changes the model's weights to teach style or behaviour. Use RAG for knowledge that changes, fine-tuning for tone or format.
Do I need a vector database for RAG? Not always. If your content fits comfortably in the model's context window, put it all in the prompt and use prompt caching. Use a vector database once the content is too large or changes often.
How do I stop a RAG chatbot from making things up? Instruct it to answer only from the retrieved sources and to say when it doesn't know, show citations, improve retrieval with hybrid search and reranking, and run an evaluation set on every change.
I build RAG chatbots and AI assistants that answer from your documents, help center or product data, with citations and access control. Tell me what it should answer and you'll get a fixed price and a monthly running-cost estimate.
Get a RAG chatbot built on your dataWritten by Antonio Kodheli
Full-stack web developer in Boston. I build modern web apps with TypeScript, Next.js, and the Claude API, and write about it here.