← Blog

Engineering note

Agentic RAG: When Vector Search Meets Agentic Reasoning

Agentic RAG: When Vector Search Meets Agentic Reasoning

Enterprise knowledge Q&A often fails not because it retrieved no relevant text, but because it retrieved several plausible answers and could not determine which one applies to this user, at this moment. Vector similarity can bring candidates into view. It cannot by itself determine whether a policy has expired, an exception overrides a rule, or a user is permitted to see a source.

That is the core question behind my report, “RAG 2026: When Vector Search Meets Agentic Reasoning.” The engineering response should not be to put every question through a multi-step agent. Keep simple questions on a measured retrieval baseline; reserve deeper investigation for conflicting sources, changing versions, and conditional answers.

A concrete failure: the right keywords, the wrong account system

An employee asks: “My company account is locked. What should I do first?” Retrieval finds two highly similar documents:

  • Group-level account self-service recovery.
  • A service-desk procedure for a local-network account.

Both contain “account,” “locked,” and “unlock.” If retrieval combines both into one context, the LLM may produce fluent instructions that target the wrong system.

Investigate this first as an evidence-selection and condition-resolution failure, not simply a hallucination. The workflow should identify the account type, user eligibility, and valid policy date. If these are missing, it should ask rather than guess.

Architecture: inexpensive recall plus conditional verification

StageCore questionMechanismEssential control
RouteIs this a verified FAQ or a complex request?Rule-first / query routingRule version and scope
RecallWhich sources might matter?BM25 + vector + RRF fusionTenant, ACL, date filters
GradeWhich candidates actually govern this question?Grading, reranking, context validationProvenance, policy version, conflicts
ResolveDoes this need another search?Bounded agent calls and rewritingCall count, time, cost, permissions
AnswerWhat can safely be said?Citation, refusal, or clarificationEvidence for each conclusion
TraceWhere did a failure occur?Retrieval and decision eventsReplayability without excess sensitive data

Rule-first does not mean bypassing governance. It uses a shorter path for conditions that have already been verified. When an exception or conflicting policy appears, the fast path must hand off or stop.

Why keep vector retrieval?

Vector search finds related meaning expressed in different words; lexical methods such as BM25 handle exact codes, document numbers, and proper nouns. A hybrid retriever is often a useful starting point, but hybrid always beats a single method is not a law. Measure recall, precision, and latency on a frozen query set.

The agent’s value is also not finding a guaranteed “single truth.” Its value is taking observable, bounded steps: locate the governing section, check effective dates, compare conflicting statements, and decide whether to answer, clarify, or refuse. When evidence is absent, the agent must be able to stop.

A shipped, first-party IT knowledge assistant

This site documents a delivered IT/process knowledge Q&A case. It is not credit adjudication, investment advice, or autonomous compliance decision-making.

The pipeline includes hybrid recall (vector, BM25, RRF), document grading, context validation, rewriting or refusal on insufficient evidence, and rule-first treatment of selected high-confidence FAQs. The numbers below are limited to its particular frozen 100-question evaluation and implementation version:

MetricReported valueInterpretation
Naive RAG87%Baseline in this test setup
Hybrid-only83.5%Ablation result, not evidence that hybrid is universally worse
Full agentic workflow98% weighted scorePartial credit may apply; not 98 fully correct answers
Strict fully correct96/100Different scoring rule from the weighted score
Unsafe outcomes0 casesWithin this frozen test set only
Full-flow latencyP95 6.19 sIncludes the measured governance path
Later rule-first pathMean 2.606 s; P95 5.636 sSeparate fast path; not the full-flow P95

These are case-specific evaluation records, not a general performance result for other enterprises or models. Any comparison must keep the test questions, grading weights, model, index, and tool versions in view.

See the Agentic RAG project page, Financial GenAI platform engineering, and Agentic AI platform contract.

When is an agent worth the extra work?

Divide requests into three practical classes:

  1. Simple lookup: Locate a document or a definition. Use lexical/vector retrieval with precise citations. Avoid gratuitous agent loops.
  2. Conditional answer: A rule applies only to certain employees or effective dates. Use metadata filters, grading, and targeted clarifying questions. A full agent loop may still be unnecessary.
  3. Cross-source research: Decompose a question, reconcile competing versions, and inspect multiple systems. A bounded agent can be valuable if every step leaves inspectable evidence.

For the third class, cap tool calls, tokens, latency, and accessible resources. If a condition can be checked deterministically, prefer that check over relying only on self-reflection by another model.

A verification chain is a diagnostic contract, not extra prompting

A useful RAG trace should let engineers answer:

  • Recall: Did the governing source enter the candidate set?
  • Permission: Was each candidate authorized for this identity at query time?
  • Version: Was a withdrawn or not-yet-effective policy used?
  • Context: Did material from another account type, product, or tenant get mixed in?
  • Answer: Do the cited spans support the final instructions? Did the system refuse unsupported claims?
  • Release: Was the updated knowledge/index regression-tested and can it be rolled back?

This separates parsing, filter, reranking, agent, and generation defects. Simply replacing the model can hide the real source of failure.

Minimum pre-release acceptance tests

RiskTest case
Old-policy contaminationCurrent and historical documents coexist; verify effective dates
Similar-system confusionMatching terms from different account or business contexts
Access-control leakCross-tenant requests, revoked access, group changes, cache isolation
Unsupported citationEvery cited span must actually support the claim
Unbounded explorationCall budgets and timeout behavior under retries
Knowledge changesFrozen regression suite, staged publication, rollback

Agentic RAG is worthwhile when it produces better evidence within explicit cost and authority boundaries. If hybrid retrieval and reranking pass the same tests, the simpler design may be the better product.

Original report PDF

The original presentation remains available for comparison with this rewritten engineering guide:

Method sources

The site’s first-party 100-question evaluation and PDF are not the experiments reported by these papers; the research lineage and case results should be interpreted separately.