RAG vs. fine-tuning for chatbots: which one are you actually doing?

A surprising number of "I want to train a custom chatbot" conversations turn out to actually be "I want to ground a chatbot in my documents." These are different things. They solve different problems. They cost wildly different amounts.

What each one actually is

Retrieval-Augmented Generation (RAG) takes your documents, chunks them, embeds them into a vector database, and at chat time retrieves the most-relevant chunks to include in the model's prompt. The model itself doesn't change — it's reading your docs through a window, every conversation.

Fine-tuning takes a base model and continues its training on your data, baking the knowledge into the model weights. The model itself changes — it has new "instincts" shaped by what you trained it on.

Use RAG when…

• Your knowledge changes often (docs get updated, products get added)
• You need provenance — "this answer came from page 7 of contract X"
• Your data isn't large enough to fine-tune well (under ~10k examples)
• You want a knob to adjust how much of a doc to include (top-K, similarity threshold)
• You don't want to retrain every time a fact changes

Use fine-tuning when…

• You want a specific style or persona (a brand voice, a domain dialect)
• You're changing model behavior, not adding facts (e.g., "always respond in JSON")
• You have a large, clean dataset of input/output pairs
• Latency matters and you don't want to retrieve documents per request
• The information is stable (a programming language's syntax, a legal jurisdiction's statutes)

The common confusion

If you say "I want to build a chatbot that answers questions about our docs," 99% of the time you mean RAG. You don't need fine-tuning. Fine-tuning would bake your docs into the model's memory, but it can't tell you which doc an answer came from, and it forgets things you updated yesterday.

If you say "I want a chatbot that talks like our brand," that's fine-tuning territory — though even there, a well-crafted system prompt with a few examples usually gets you 80% of the way there at zero training cost.

What Ashh.ai does

RAG, by default, with HNSW-indexed pgvector on the back end. You upload docs, we chunk and embed (nomic-embed-text, 768-dim, on our GPUs), and at chat time we retrieve top-K chunks for the model. No fine-tuning required, no training cost, knowledge updates are instant (re-embed the changed file, that's it).

If you genuinely need fine-tuning — domain-specific voice, a unique reasoning pattern — that's a custom engagement we can scope. But it's rarely the right answer for "chat with my docs."