Agentic RAG 的軌跡評估
Series · 1 posts
-
Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation
AdvancedA deep reading of Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation: turning intermediate answer quality, utility, and trajectory signals into an early-stopping controller, then checking the savings and transfer boundary.
Understand it in 90 seconds
- Problem
- An agentic RAG system may execute several query, retrieve, read, and reasoning rounds before it produces a final answer. Running every instance to natural stopping wastes cost after the answer has saturated, while continuing after a failure pattern may only repeat the failure. The paper asks whether each partial answer can provide a prediction of quality and utility.
- Core insight
- Partial quality and incremental utility are different targets. Quality asks how close the current answer is to the gold answer; utility asks how much quality changed from the previous round. Quality is easier to predict, while the sign and magnitude of utility depend more strongly on trajectory, retrieval noise, and task.
- Strongest evidence
- On Search-R1 and R1-Searcher over HotpotQA, 2WikiMultiHopQA, and MuSiQue, the authors compare supervised and unsupervised predictors. Search-R1 reaches about 0.438 at the best quality Pearson correlation and about 0.321 for utility; with thetaP=0.3 and thetaU=0.2, the controller reduces average iterations from 3.21 to 2.86 and preserves 97.60% of natural-stopping quality.
- Main boundary
- Partial answer quality is probed with F1 against a gold answer. That is not open-ended answer quality, and it does not show that a threshold transfers to a new retriever, model, corpus, answer format, or controller. Probing itself has generation cost.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact