What problem does RAG solve that fine-tuning does not?

RAG and fine-tuning solve different problems. Fine-tuning adjusts the model's weights to change its behavior or improve performance on a specific task, but does not reliably inject factual knowledge. A fine-tuned model may hallucinate about specific facts it was fine-tuned on. RAG retrieves the specific document at inference time and puts it in the prompt, so the model is grounding its answer in the retrieved text. RAG is better for factual question answering over a document corpus. Fine-tuning is better for changing response style, format, or task-specific behavior.

What is a vector database and do you always need one for RAG?

A vector database stores embeddings (numerical representations of text) and supports similarity search: given a query embedding, find the documents whose embeddings are most similar. Popular options include Pinecone, Weaviate, Chroma, and pgvector for PostgreSQL. You do not always need a dedicated vector database. For small document collections (fewer than 100K documents), pgvector in PostgreSQL provides adequate similarity search performance without adding a new infrastructure component. Dedicated vector databases become relevant when the document count grows large, when you need filtered similarity search, or when search latency is a primary constraint.

What is chunking and why does it matter for RAG quality?

Chunking is the process of splitting documents into smaller pieces before embedding them. LLMs have context length limits, so entire documents often cannot fit in a single prompt. Chunking strategy significantly affects retrieval quality: chunks that are too small lose context and produce incomplete answers; chunks that are too large exceed context limits and dilute relevance. Effective chunking strategies preserve semantic coherence: splitting at paragraph or section boundaries rather than at fixed character counts, and including overlap between chunks so context is not lost at chunk boundaries.

When does RAG produce poor results despite being implemented correctly?

RAG produces poor results when the document corpus does not contain the answer (the model cannot retrieve what is not there), when documents are poorly structured or inconsistently formatted (making embedding quality low), when the retrieval step returns irrelevant documents because the query and document language differ (the user asks 'how do I reset my password' but documentation says 'account recovery'), and when retrieved context is accurate but incomplete (requiring multi-hop reasoning across multiple documents). Each of these failure modes requires a different fix.

How do you evaluate RAG quality systematically?

RAG quality has two separable components: retrieval quality (are the right documents being retrieved?) and generation quality (given the retrieved documents, is the answer correct?). Evaluate retrieval quality by measuring recall@k: for a set of test questions with known relevant documents, what fraction of those documents appear in the top k retrieved results? Evaluate generation quality using LLM-as-judge evaluation or human annotation against a test set of questions and expected answers. Measuring both components independently makes it possible to diagnose whether quality issues stem from retrieval or generation.

AI Integration and Vibe Coding RescueMay 23, 20266 min read

RAG (Retrieval Augmented Generation) for SaaS: When It Helps and When It Does Not

Retrieval Augmented Generation (RAG) is an AI architecture pattern that combines a retrieval system (typically vector search over a document store) with a large language model. When a user asks a question, the system retrieves relevant documents from the store and includes them in the LLM prompt as context, allowing the model to generate answers grounded in specific documents rather than relying on training data alone. RAG is used when the answer depends on information that is proprietary, recent, or not in the LLM's training data.

Written by Yashveer Singh, founder of Yashveer Labs.

What you need to know

RAG is the right architecture when the answer depends on information not in the LLM's training data: proprietary documentation, recent events, company-specific knowledge.
RAG is the wrong architecture when the task is reasoning, code generation, summarization of provided text, or any task that does not require external factual lookup.
Retrieval quality determines answer quality. A RAG system with good generation but poor retrieval produces confident-sounding wrong answers.
Chunking strategy is the most underestimated implementation decision. Poor chunking degrades retrieval quality regardless of embedding model quality.
Start with a small, well-curated document corpus before scaling. A RAG system with 100 high-quality documents outperforms one with 10,000 poorly processed documents.

The core argument

RAG became the default recommendation for any AI feature involving proprietary knowledge. That created a generation of RAG implementations that are deployed where they should not be. A product that needs to summarize a user-provided document does not need RAG. A chatbot that answers questions about product features might need RAG, but only if those features change faster than model fine-tuning cycles. An AI assistant that answers support questions about a specific customer's account configuration absolutely needs RAG, because that information cannot be in any model's training data.

The distinction matters because RAG adds significant complexity. The system now has two quality surfaces to maintain: the document corpus (which must be current, accurate, and well-processed) and the generation layer (which must produce coherent answers from retrieved context). Teams that add RAG without this complexity being necessary end up maintaining infrastructure that does not improve the product and introduces new failure modes like outdated documents producing incorrect answers with high confidence.

When RAG is the right choice, the implementation decisions that matter most are: the chunking strategy for the document corpus, the embedding model used for both documents and queries, the retrieval strategy (dense retrieval, sparse retrieval, or hybrid), and the prompt structure that presents retrieved documents to the model. In my experience working on knowledge base search features for Velmora, the chunking strategy had more impact on answer quality than the choice of embedding model. Semantic chunking that preserved paragraph boundaries outperformed fixed-size chunking by a significant margin on the evaluation set.

Common mistakes

Using RAG for tasks that do not require factual retrieval. A summarization feature that summarizes a document the user just uploaded does not need to retrieve anything. The document is already in the context. Adding retrieval to tasks that do not benefit from it adds latency, cost, and complexity for no quality improvement.

Not building an evaluation set before implementing. RAG quality is hard to assess subjectively. "The answers look better" is not a reliable signal. Build a test set of 20 to 50 representative questions with expected answers before implementing, and use it to measure quality changes objectively as the system evolves.

Using fixed-size character chunking. Splitting documents every 500 characters produces chunks that cut sentences in the middle, split code blocks, and separate headings from their content. Semantic chunking that respects document structure (paragraphs, sections, list items) produces better embeddings and better retrieval.

Not handling retrieval failures gracefully. When the retrieval step returns no relevant documents (the question is outside the corpus), the model should acknowledge this rather than hallucinate an answer. The prompt must include instructions for handling low-relevance retrieved context. A RAG system without this handling produces confident wrong answers when the answer is not in the corpus.

Embedding queries and documents differently. Query text and document text often have different linguistic patterns. A question and a document paragraph that cover the same topic may have low cosine similarity if one is phrased as a question and the other as a statement. Embedding models trained specifically for asymmetric retrieval (query-to-document) address this. Using a symmetric embedding model produces lower recall on question-answering tasks.

Where to start

Define the specific knowledge gap the RAG system will fill. Write down what information is not in the LLM's training data that the system needs to answer correctly. If this list is empty or trivially small, RAG is not necessary. If the list contains substantial proprietary or frequently updated information, RAG is justified.

Build the document corpus before the retrieval pipeline. Identify the source documents, process them into clean text, and assess the quality of the processed content manually before writing any embedding code. A corpus of 50 clean, well-structured documents is better than 500 documents with inconsistent formatting and duplicate content.

Implement retrieval and generation as separable components with independent evaluation. Log the retrieved documents alongside the generated answer for every query. This allows debugging retrieval quality and generation quality independently. When an answer is wrong, the log shows whether the right documents were retrieved (generation failure) or not (retrieval failure), which determines the fix.

Frequently asked

Author

The reason I write these

I write these because the writing is the proof. Yashveer Singh, founder of Yashveer Labs. The systems I build are not theoretical. They are running right now, serving real users, generating real revenue. That is the bar I hold this writing to. If you want to hire someone who can match that bar, I am the call.

Start the conversation See the work DM on Instagram

RAG (Retrieval Augmented Generation) for SaaS: When It Helps and When It Does Not

What you need to know

The core argument

Common mistakes

Where to start

Related reading

Frequently asked

The reason I write these

Multi Agent Systems for SaaS: A Practical Architecture

The AI Onboarding Assistant: A High Value SaaS Feature

The Difference Between an AI Wrapper and an AI Product

Streaming AI Responses to Users: An Architecture Primer