REALM deep reading

Series · 1 posts

← Back to Paper Reading

  • REALM: Wire Retrieval into LM Pre-Training, but Do Not Treat Joint Training as a Ready-Made RAG Stack

    Intermediate Retrieval, memory, and production RAG
    REALM deep reading: Part 1 , Note: Aug 27, 2026 , Paper: 2020 , NLP

    A source-grounded reading of Guu et al., ICML 2020: a differentiable knowledge retriever is pre-trained with an MLM signal, an asynchronously refreshed MIPS index, and Open-QA fine-tuning. With CC-News / Wikipedia, NQ Exact Match is 40.4, above ORQA and T5-11B. This is costly retrieval-augmented pre-training—not Lewis RAG generation and not DPR’s cheaper dual-encoder recipe.

    Understand it in 90 seconds
    Problem
    Pre-trained LMs store world knowledge in parameters; covering more facts pushes networks ever larger, and the stored knowledge is hard to locate or update.
    Core insight
    During pre-training, add a learnable knowledge retriever that fetches documents $z$ from a corpus such as Wikipedia, backpropagate an MLM signal through retrieval (treating $z$ as a latent variable), and use asynchronous MIPS refreshes so the index can keep up with changing document embeddings.
    Strongest evidence
    ICML Table 1 Open-QA Exact Match—REALM with $X$=CC-News and $Z$=Wikipedia reaches NQ 40.4, WQ 40.7, CT 42.9; same-scale ORQA scores 33.3 / 36.4 / 30.1; T5-11B (11318M) reaches only 34.5 on NQ. Table 2: 30× stale MIPS drops NQ-dev Exact Match to 28.7.
    Main boundary
    Memory is the 20 Dec 2018 English Wikipedia dump (just over 13 million chunks of up to 288 wordpieces); evaluation is English Open-QA with extractive spans; training needs 64-TPU pre-training and periodic index rebuilds; this is not production RAG, not generative RAG, and not when-to-retrieve.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact