Enterprise RAG Guide: Retrieval Architecture, Evaluation, and Production Delivery

Enterprise RAG is not about letting a model see more documents. It must deliver traceable evidence to the right identity at the right time and measure whether that evidence improves the answer. A basic chunk–embed–vector-search pipeline quickly runs into permissions, versions, tables, multi-hop questions, and failures that are impossible to diagnose.

This hub guide treats RAG as a governed knowledge supply chain, from ingestion, indexing, retrieval, and reranking through context assembly, generation, evaluation, and operations.

1. How is enterprise RAG different?

Enterprise settings add at least five requirements:

• Permission consistency: results must honor user, group, tenant, and source-system access.
• Version and freshness: obsolete, duplicate, or withdrawn documents must stop affecting answers.
• Traceability: answers identify the exact passage, version, and ingestion time used.
• Evaluation: teams can distinguish missing retrieval, bad ranking, context failure, and generation error.
• Operations: latency, cost, update frequency, and service levels are explicit trade-offs.

Enterprise RAG therefore sits at the intersection of search, data engineering, LLM applications, security, and platform operations.

2. The complete RAG pipeline

1. Sources and ingestion

Create a source inventory with ownership, sensitivity, update method, usable fields, and deletion behavior. Preserve source URL, document ID, version, timestamp, and ACL at ingestion. Governance is difficult to reconstruct after those fields are lost.

2. Parsing and chunking

Chunking should follow document structure and use. Titles, sections, tables, code, and page locations belong in metadata. Fixed token windows are only a baseline: chunks that are too small lose context, while oversized chunks reduce precision and waste tokens.

3. Indexing

Enterprise search usually keeps lexical and vector indexes. Lexical retrieval handles product codes, exact strings, and proper nouns; vector retrieval handles semantic similarity across different wording. Add a knowledge graph only when structural relations or multi-hop questions justify it.

4. Query understanding and retrieval

Resolve language, abbreviations, time range, and required filters before searching. Hybrid search should fuse results with a measurable method, not simply add retrievers. Query rewriting must preserve the original intent rather than turn the request into a different question.

5. Reranking and context assembly

A reranker addresses relevant documents in the wrong order. Context assembly deduplicates passages, covers distinct subquestions, respects the token budget, and preserves citations. A larger top-k can reduce quality by burying the strongest evidence in noise.

6. Generation and answer policy

The prompt should require answers from permitted evidence, preserve citations, and state when evidence is missing or contradictory. High-risk domains may need rule-based checks or human approval after generation.

7. Feedback and evaluation

Within privacy constraints, retain retrieved candidates, reranking scores, final context, citations, and feedback. Without this trace, changing a model, index, or prompt cannot be tied to a specific improvement.

3. Choosing a RAG architecture

Architecture	Best fit	Primary cost
Vector RAG	Semantic questions over relatively uniform content	Can miss exact terms and filters
Hybrid RAG	Most enterprise document and search use cases	Requires fusion, filtering, and reranking
GraphRAG	Dense relationships, multi-hop reasoning, global summaries	Higher graph construction and update cost
Agentic RAG	Dynamic source selection, decomposition, iterative verification	Higher latency, cost, and path variability
Multimodal RAG	Evidence lives in figures, layout, or images	More complex parsing, indexing, and evaluation

The default should be hybrid retrieval with a reranker. Add graph, agent, or multimodal capabilities only when evaluation proves that relationships, cross-source planning, or visual evidence are the bottleneck.

4. What should RAG evaluation measure?

Do not reduce quality to whether an answer “looks good.” Evaluate five layers:

1. Retrieval: Recall@k, MRR, nDCG, and whether required evidence enters the candidate set.
2. Context: relevance, completeness, duplication, and token-budget efficiency.
3. Answer: correctness, faithfulness, completeness, citation coverage, and refusal quality.
4. Security: ACL leakage, cross-tenant access, sensitive content, and prompt-injection resistance.
5. Operations: P50/P95 latency, query cost, index freshness, and error rate.

Sample evaluation cases from real work. Include direct lookup, multi-hop, time-sensitive, table, same-name entity, unanswerable, and unauthorized questions. LLM judges can assist subjective scoring, but humans must calibrate samples and high-risk cases.

5. Access control and knowledge governance

Apply ACLs during retrieval instead of fetching everything and asking the model to ignore forbidden content. Indexes must synchronize deletions and permission changes. Cache keys must include tenant and authorization scope. Citation pages must authorize again so a safe answer cannot link to an exposed source.

Every chunk should retain source, version, owner, ingestion time, expiry, language, structural location, and allowed principals. Highly sensitive collections may require separate indexes, encryption keys, or service boundaries.

6. Diagnosing common failures

Symptom	Inspect first	Common direction
A known document is absent	Ingestion, parsing, filters, freshness	Repair the data pipeline before the prompt
Relevant content is found but the answer is wrong	Reranking, context, generation trace	Improve reranking, deduplication, or policy
Proper nouns are frequently missed	Lexical hits and query normalization	Add hybrid retrieval, dictionaries, exact filters
Multi-hop answers are incomplete	Subquestion and source coverage	Decomposition, graph, or agentic RAG
Citations use an obsolete version	Version metadata and deletion sync	Add validity windows and authoritative-source rules
Latency or cost is excessive	Stage timing and token use	Cache, reduce candidates, parallelize, route models

Find the failing layer before changing its component. A larger model often hides a data or retrieval defect while raising cost.

7. A path from PoC to production

1. Select one knowledge domain and 50–200 representative questions.
2. Establish lexical, vector, and hybrid baselines; measure retrieval and answers separately.
3. Connect source versions and ACLs; test deletion, changes, and tenant isolation.
4. Add reranking, citations, and abstention when evidence is insufficient.
5. Classify errors through traces and upgrade architecture only for proven bottlenecks.
6. Set launch gates for quality, P95 latency, cost per query, and index freshness.

8. Topic reading path

Read the cluster in this order:

1. Agentic RAG: Vector Search Meets Agent Reasoning
2. PixelRAG: Visual Evidence for Complex Documents
3. Open Knowledge Format: Portable, Governed Knowledge
4. Graph RAG and LLMs: Relations and Multi-Hop Retrieval
5. Financial GenAI Platform Engineering: RAG Governance and Operations
6. LangChain OpenWiki: Building Retrieval from Open Knowledge

See the full delivery context in the Agentic RAG enterprise knowledge assistant case study, validated at 98% weighted accuracy and 2.6 seconds average latency. For systems that act beyond retrieval, continue with the AI Agent guide.

9. Limits and trade-offs

RAG cannot turn low-quality, contradictory, or ungoverned knowledge into reliable truth, nor guarantee that a model reasons correctly from good evidence. More complex architectures cover more cases but increase update, evaluation, and operational cost.

The sustainable approach is to establish data, permissions, baselines, and evaluation first, then use observed failures to justify GraphRAG, agentic RAG, or a larger model.