Financial Retrieval Evaluation

Series · 1 posts

← Back to Paper Reading

  • FinRank: Hard-Negative Retrieval Evaluation for Financial-Document RAG

    Intermediate Retrieval, memory, and production RAG
    Financial Retrieval Evaluation · Part 1 · Note · Aug 11, 2026 · Paper · 2026 · NLP

    A deep reading of FinRank: how company, year, and disclosure boundaries create deceptively plausible evidence, and why pooled retrieval, hard negatives, and metadata filters must be evaluated together.

    Understand it in 90 seconds
    Problem
    Financial-document QA fails for more than one reason. The same term can appear in different companies, filing years, and notes, while a metadata filter can remove the true supporting passage. FinRank asks whether a retriever can preserve the right evidence among these plausible but wrong alternatives.
    Core insight
    The authors build 1,185 human-authored financial QA records, a 5,230-passage pooled corpus, and 6,021 curated hard negatives, then test generalization across multiple splits, query rewrites, and provenance boundaries.
    Strongest evidence
    On the pooled corpus, the reported Recall@10 is 44.8 for e5-mistral-7b-instruct and 32.1 for BM25. Metadata-filtered BM25 reaches 55.0, but Section 7.1 notes that first-occurrence metadata can exclude the gold passage. In Section 7.3, curated hard negatives lower pairwise accuracy by 13.0–20.5 percentage points versus random negatives.
    Main boundary
    The paper does not measure answer generation, citation correctness, or faithfulness. The data is concentrated in U.S. 10-K/10-Q filings from 2024–2025, with annotation and distribution caveats. This is a strong retrieval-evaluation starting point, not proof of safe financial RAG deployment.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact