Sequence Modeling Foundations
4 reading notes
Build a foundation for reading sequence-transduction papers through classic encoder–decoder and self-attention architectures.
Reader question
Where does a sequence model change its control point, and which evidence still transfers to today’s systems?
Reading library
Read this topic in depth
Every paper reading currently filed under this research topic.
-
Transformer: Drop Recurrence for Self-Attention, but WMT 2017 BLEU Does Not Represent Later LLMs
Intermediate Build the foundations firstA source-grounded reading of Vaswani et al., NeurIPS 2017 / arXiv:1706.03762: stacked encoder-decoder with multi-head self-attention and positional encodings replaces RNNs and convolutions for machine translation. On WMT 2014, the big model reaches 28.4 BLEU EN-DE and 41.8 BLEU EN-FR; this is 2017 sequence-transduction evidence, not a BERT, GPT-3, or ViT contract.
Understand it in 90 seconds
-
InstructGPT: Align Instructions with Human Feedback, but 2022 Preference Win Rates Do Not Represent Later ChatGPT
Intermediate Build the foundations firstA source-grounded reading of Ouyang et al., NeurIPS 2022 / arXiv:2203.02155: while retaining the GPT-3 architecture and updating its weights, a three-stage pipeline—SFT demonstrations, a reward model, and PPO (default PPO-ptx)—aligns a pretrained LM to human preferences. 175B InstructGPT is preferred over 175B GPT-3 85±3% of the time; this is 2022 API labeler evidence, not a ChatGPT product SLA, GPT-4 eval, or DPO contract.
Understand it in 90 seconds
-
Speculative Decoding: Draft with a Small Model, Verify in Parallel, but T5 Speedups Do Not Represent Every Serving Stack
Intermediate Build the foundations firstA source-grounded reading of Leviathan et al., ICML 2023 / arXiv:2211.17192: a cheap draft model M_q proposes tokens, the target model M_p verifies a chunk in parallel, and speculative sampling keeps the output distribution identical to target-only decoding. T5-XXL 11B reaches 2.3X-3.4X wall-clock speedup on T5X; this is 2023 lossless inference-algorithm evidence, not a GPTQ, FlashAttention, vLLM, Medusa, or EAGLE contract.
Understand it in 90 seconds
-
RAG: Attach Retrieval to Generation, but Do Not Treat 2020 RAG as a Production RAG Platform
Intermediate Retrieval, memory, and production RAGA source-grounded reading of Lewis et al., NeurIPS 2020: BART is paired with a dense retriever over Wikipedia, and RAG-Sequence / RAG-Token condition generation on retrieved passages. RAG-Seq reaches 44.5 Exact Match on NQ; this is a 2020 method paper, not a 2025 production RAG platform and not an agent loop.
Understand it in 90 seconds
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact