RAG begins with a retrieval-and-answering task whose sources can be inspected, not with choosing a vector database. Splitting files, embedding them and storing vectors covers only part of the system. This guide uses a fictional community bicycle manual library to connect concepts, a small working knowledge base and retrieval improvement. It is a software-learning example; real repair decisions and safety issues still belong with qualified people rather than an answer generator.
Scenario one: understand indexing and answering as separate paths
The indexing path converts authorized documents into text, cleans and splits them, preserves metadata such as title, revision, page and access scope, and builds retrieval indexes. The answering path takes a user's question, finds evidence the user may access, ranks and assembles it, then generates a supported answer. RAG is generally not model retraining. Document changes require index and cache updates rather than an assumption that the model will automatically remember new content.
For “Where is the maintenance interval documented for this folding bicycle?”, the result should identify the matching model and revision. A semantically similar paragraph about another model is not interchangeable evidence. When the documents do not answer a question, say so and optionally suggest a search direction instead of turning model memory into a manual requirement. A citation's existence does not establish support; inspect its content and scope.
Start with a small authorized sanitized collection and a question set a person can review. Include exact terms, paraphrases, model identifiers, expired information, missing answers and inaccessible sources. Record relevant documents and acceptable conclusions for each query. This helps expose pipeline failures; it is not a complete production acceptance suite. Access, deletion and versioning matter before a collection reaches millions of vectors.
Scenario two: build a knowledge base with inspectable artifacts
Create a manifest containing document_id, source, permitted use, revision, effective date, access label, file digest and ingestion state. Inspect extracted text before indexing, especially scans, tables, headers and multicolumn layouts. A table whose units or column headings disappeared cannot be repaired by adding more vectors. Record failed files explicitly instead of silently skipping them.
Split initially along headings and meaningful boundaries, retaining conditions with the conclusions they qualify. Use limited overlap or parent context where it helps. Short chunks may lose conditions; long chunks may introduce irrelevant content. Chunk size is an experiment parameter, not a universal answer. Bind chunk identifiers to document revision and location, and record parser, splitting rule and embedding model versions for reproducibility and deletion.
Query and document embeddings need compatible models and preprocessing. A model change can alter dimension, vector space and score interpretation, so build a new index revision and evaluate it instead of mixing old and new vectors. A teaching baseline may use simple local storage or lexical search. Real selection also depends on access filters, update frequency, persistence, operations, latency and budget, not just document count.
Haystack is an Apache-2.0 open-source framework worth evaluating for reusable indexing, retrieval and generation components rather than assembling every interface from scratch. Check the chosen version, integrations, model service and maintenance requirements. This article has not installed or deployed it, and a tutorial pipeline is not production acceptance. Framework components and Skill guidance do not replace actual authorization and checks.
For the first query, inspect controlled candidate identifiers and sources before trusting the final answer. Check ranking and the assembled context. Only authorized evidence should reach the generator; do not expose private documents and merely instruct the model not to reveal them. Treat retrieved text as evidence rather than new authority. Ask the generator to distinguish unsupported and conflicting information and verify that key claims really follow from the cited passages.
Worked example: diagnose an answer that exists but is missed
Suppose a query asks for the F-27 model's maintenance appendix but returns generic advice. Check whether the appendix was ingested and parsed, whether F-27 was corrupted during extraction, whether filtering removed it, whether its chunk entered the candidate set, whether reranking displaced it and whether context assembly dropped the title. Making the final answer longer is not the first repair.
If the correct passage is retrieved but the response uses the wrong model, investigate context or generation constraints. If it never enters the candidate set, investigate ingestion or retrieval. Keep query identifier, input revision, candidate ranks, selected context and output with a failure category. After changing splitting or retrieval, rerun other models' questions so a narrow fix does not damage ordinary cases.
A bounded agent task is: “Use only files listed under approved-manuals. Inspect extraction and source metadata, then create a removable test index. For query-set-v1 return candidate chunks, citations and reasons for unanswered questions. Do not upload unauthorized files, connect to a production database or give unsupported repair instructions. Report executed steps and failures without claiming unrun evaluations passed.”
Scenario three: compare retrieval improvements against a baseline
Establish a simple baseline before comparing lexical, vector and hybrid retrieval. Identifiers and exact terms often need literal matching, while paraphrases may benefit from semantic similarity, but neither route is infallible. Hybrid retrieval combines candidate paths; a method such as reciprocal rank fusion combines rankings, and a reranker then assesses candidate relevance. Decide whether it helps using the project's questions, latency and cost. Small data is not a universal reason to skip it, and an advanced name is not a reason to install every strategy.
Separate retrieval and answer evaluation. Retrieval asks whether relevant passages appear in the first k results and where they rank. Interpret complete recall@k only when the relevant set is sufficiently annotated, stating limits when judgments are incomplete. Answer evaluation checks support, citation applicability, conflict handling, abstention and usability. Report languages, identifiers, cross-section questions and no-answer cases separately rather than hiding them in one average accuracy score.
When changing chunking, top_k, reranking or compression, compare one main variable with model, index and test set fixed. More context can increase noise and cost. Compression can remove negation or units. Multi-hop retrieval adds work and opportunities for error. Diagnose the failure before adding complexity and keep a rollback configuration. Model-generated questions and self-grading alone are not enough; have people review important cases and counterexamples.
Test updates and deletion: new revisions are found, old ones are not mistaken for current guidance, and withdrawn passages and caches no longer answer queries. Logs, backups and external retention policies need separate treatment; deleting one vector is not proof that every copy was erased. Test different identities asking the same question and inspect the sources available to each, not just whether the final response omitted a keyword.
Verify the interface with a matching Skill
For a running local Q&A page, Webapp Testing fits the frontend checkpoint: submitting a query, loading, empty results, error states and source links. It needs Python, Playwright and browser dependencies, startup instructions, a port and authorized test data. Inspect the rendered DOM, choose stable controls and retain screenshots and logs. This does not establish retrieval quality, backend deletion, access isolation or factual correctness; those need evidence from the corresponding parts of the system.
The six RAG names in the source article do not have matching verified SkillPI source records in this review, so no installation links are guessed. For future candidates check fixed source, license, model and store dependencies, file and network access and actual output before testing them on the same sanitized manuals. Installation does not prove invocation, and a chat page does not establish a production release.
Troubleshooting and FAQ
Should I study databases first? Understand the full path and inspect a small task first, while treating access, persistence and updates as early requirements. How long will a build take? Formats, environment, source quality and acceptance scope determine the work; there is no universal afternoon guarantee. What if retrieval succeeds but the answer is wrong? Check support, revision mixing, selected context and generation beyond evidence.
Does RAG eliminate hallucination? It does not; it supplies retrievable evidence alongside checks and abstention. Why did changing embeddings reduce quality? Check index compatibility, dimensions, preprocessing and query settings, then evaluate a new revision. Does a passed UI test permit release? Source rights, authorization, updates, deletion, resource limits and failure recovery still need review. Deliver the manifest, versioned index notes, query set, retrieval and answer evaluations and pending checks so another person can reproduce what was actually established.