REALM deep reading
Series · 1 posts
-
REALM: Wire Retrieval into LM Pre-Training, but Do Not Treat Joint Training as a Ready-Made RAG Stack
Intermediate Retrieval, memory, and production RAGA source-grounded reading of Guu et al., ICML 2020: a differentiable knowledge retriever is pre-trained with an MLM signal, an asynchronously refreshed MIPS index, and Open-QA fine-tuning. With CC-News / Wikipedia, NQ Exact Match is 40.4, above ORQA and T5-11B. This is costly retrieval-augmented pre-training—not Lewis RAG generation and not DPR’s cheaper dual-encoder recipe.
Understand it in 90 seconds
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact