Agentic RAG Failure Diagnosis and Attribution
Series · 1 posts
-
When Failure Propagates, Can We Still Find the Start? Causal Failure Attribution in Agentic RAG
AdvancedA deep reading of When Failures Propagate: an interventional benchmark, three-hop MuSiQue, and certified content corruption that separate failure detection, causal attribution, propagation, and recovery in agentic RAG.
Understand it in 90 seconds
- Research problem
- Agentic RAG splits retrieval, reasoning, and answering across several hops. An early bad piece of evidence can become query drift, a wrong bridge, or a wrong answer, but later retrieval can also repair it. A final answer or the last trace state therefore cannot directly tell us which hop first caused the failure.
- Core insight
- AgenticRAG-FP injects a certifiable fault at a specified hop, then resumes execution from the changed prefix and regenerates the suffix. The diagnoser is not guessing the cause of a static error trace; it is tested against the known injectedathop label with exact-hop attribution.
- Strongest evidence
- The strict dense Claude Haiku 4.5 sweep uses 80 three-hop MuSiQue questions. Among cases that still fail, coverage-based exact-hop accuracy is hop 1: 0.91 [0.81, 0.98], hop 2: 0.00 [0.00, 0.00], and hop 3: 0.00 [0.00, 0.00], with failed denominators 43, 36, and 21 respectively (Table 2, Section 7.1).
- Main boundary
- The result supports the claim that, under this strict intervention and suffix-resumption setup, coverage’s hop-level signal disappears at deeper hops. It does not support the claim that every natural agentic RAG failure is un-attributable. The content study has only 18 hop-2 failed cases and 3 hop-3 failed cases, so its method comparisons cannot be generalized.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact