RAG deep reading

Series · 1 posts

← Back to Paper Reading

  • RAG: Attach Retrieval to Generation, but Do Not Treat 2020 RAG as a Production RAG Platform

    Intermediate Retrieval, memory, and production RAG
    RAG deep reading: Part 1 , Note: Aug 27, 2026 , Paper: 2020 , NLP

    A source-grounded reading of Lewis et al., NeurIPS 2020: BART is paired with a dense retriever over Wikipedia, and RAG-Sequence / RAG-Token condition generation on retrieved passages. RAG-Seq reaches 44.5 Exact Match on NQ; this is a 2020 method paper, not a 2025 production RAG platform and not an agent loop.

    Understand it in 90 seconds
    Problem
    Large pretrained models store facts in their parameters, yet still lag task-specific architectures on knowledge-intensive work; parametric memory is hard to update, hard to inspect, and prone to hallucination.
    Core insight
    Pair a pretrained seq2seq generator (BART) with a pretrained dense retriever (initialized from DPR) over a Wikipedia index. The decision point moves from “answer from parameters only” to “retrieve passages, then condition generation.” RAG-Sequence shares one document across the sequence; RAG-Token may switch documents per token.
    Strongest evidence
    Table 1 open-domain QA: on NQ, RAG-Seq 44.5 and RAG-Token 44.1 beat DPR 41.5, REALM 40.4, and T5-11B+SSM 36.6. Table 2 generation and classification: on Open MS-MARCO, RAG-Seq is +2.6 Bleu and +2.6 Rouge-L versus BART; on FEVER-3, 72.5 sits 4.3 points below the then pipeline SOTA of 76.8, with no intermediate retrieval supervision.
    Main boundary
    The memory is the December 2018 Wikipedia dump split into 21M 100-word chunks, not a private corpus; retrieval is dense MIPS, not a production hybrid; there is no agentic search / read / final loop, and no 2026 enterprise sense of citation faithfulness.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact