Sequence Modeling Foundations

4 reading notes

← Back to Paper Reading

Build a foundation for reading sequence-transduction papers through classic encoder–decoder and self-attention architectures.

Reader question

Where does a sequence model change its control point, and which evidence still transfers to today’s systems?

Reading library

Read this topic in depth

Every paper reading currently filed under this research topic.

  • Transformer: Drop Recurrence for Self-Attention, but WMT 2017 BLEU Does Not Represent Later LLMs

    Intermediate Build the foundations first
    Original Transformer deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2017 , NLP

    A source-grounded reading of Vaswani et al., NeurIPS 2017 / arXiv:1706.03762: stacked encoder-decoder with multi-head self-attention and positional encodings replaces RNNs and convolutions for machine translation. On WMT 2014, the big model reaches 28.4 BLEU EN-DE and 41.8 BLEU EN-FR; this is 2017 sequence-transduction evidence, not a BERT, GPT-3, or ViT contract.

    Understand it in 90 seconds
    Problem
    Before 2017, state-of-the-art sequence transduction (seq2seq) relied on RNN/LSTM/GRU encoder-decoders whose computation unfolds along time steps, limiting within-example parallelism on long sequences. Bahdanau et al. already attached attention to RNNs, but recurrence remained the backbone (Sections 1-2).
    Core insight
    The Transformer stacks multi-head self-attention and position-wise FFN in both encoder and decoder, injects order with sinusoidal positional encodings, and removes recurrence and convolutions entirely. The control point is parallel global attention versus sequential hidden states; path length for long-range dependencies is $O(1)$ per layer (Table 1).
    Strongest evidence
    WMT 2014 newstest2014 (Table 2): Transformer (big) reaches 28.4 BLEU EN-DE (above prior bests including ensembles) and 41.8 BLEU EN-FR; big-model training takes 3.5 days on 8xP100 GPUs (300K steps). The base model reaches 27.3 BLEU EN-DE with training FLOPs of $3.3\times10^{18}$, below GNMT+RL at $2.3\times10^{19}$. Hardware text: base training 12 hours / 100K steps at 0.4 seconds per step (Section 5.2).
    Main boundary
    The task is supervised MT encoder-decoder, not a pretrained language model, not bidirectional BERT, not decoder-only GPT, and not ViT. BERT, GPT-2/3, T5, LLaMA, and ChatGPT benchmarks are not in this PDF; YOLO VOC mAP and ResNet ImageNet 4.49% are not MT contracts either.
    Read the full deep dive

  • InstructGPT: Align Instructions with Human Feedback, but 2022 Preference Win Rates Do Not Represent Later ChatGPT

    Intermediate Build the foundations first
    Original InstructGPT deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2022 , NLP

    A source-grounded reading of Ouyang et al., NeurIPS 2022 / arXiv:2203.02155: on a frozen GPT-3 architecture, a three-stage pipeline—SFT demonstrations, a reward model, and PPO (default PPO-ptx)—aligns a pretrained LM to human preferences. 175B InstructGPT is preferred over 175B GPT-3 85±3% of the time; this is 2022 API labeler evidence, not a ChatGPT product SLA, GPT-4 eval, or DPO contract.

    Understand it in 90 seconds
    Problem
    Bigger LMs do not automatically follow user intent better. GPT-3-class models under a next-token pretraining objective often produce untruthful, toxic, or unhelpful outputs (Section 1). Language modeling is misaligned with responding helpfully and safely to instructions.
    Core insight
    On a frozen GPT-3 architecture, run three stages: (1) SFT—fine-tune on demonstrations from about 40 contractors; (2) RM—train a 6B reward model on human rankings; (3) PPO—optimize the policy against RM scores with a per-token KL penalty toward SFT; the default PPO-ptx mixes pretraining gradients to reduce public NLP regressions (Figure 2, Equation 2, Section 3.5). The control point is human-preference alignment, not a new attention stack.
    Strongest evidence
    Labeler preference evals on the API prompt distribution (Figure 1, Section 4.1): 175B InstructGPT is preferred to 175B GPT-3 85±3% of the time; to few-shot GPT-3 71±4%; 1.3B InstructGPT still beats 175B GPT-3 (>100× parameter gap). Versus the 175B SFT baseline, InstructGPT wins 73.4±2%, beating FLAN/T0 fine-tunes (26.8±2%, 29.8±2%).
    Main boundary
    Closed 2022 GPT-3 family models; preferences come from a specific labeler pool and API Playground distribution (Sections 5.2–5.3). ChatGPT product metrics, GPT-4, DPO, Llama-2-chat, and Constitutional AI are outside this PDF; YOLO mAP and Transformer WMT BLEU are not alignment contracts.
    Read the full deep dive

  • Speculative Decoding: Draft with a Small Model, Verify in Parallel, but T5 Speedups Do Not Represent Every Serving Stack

    Intermediate Build the foundations first
    Original Speculative Decoding deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2023 , NLP

    A source-grounded reading of Leviathan et al., ICML 2023 / arXiv:2211.17192: a cheap draft model M_q proposes tokens, the target model M_p verifies a chunk in parallel, and rejection sampling keeps the output distribution identical to target-only decoding. T5-XXL 11B reaches 2.3X-3.4X wall-clock speedup on T5X; this is 2023 lossless inference-algorithm evidence, not a GPTQ, FlashAttention, vLLM, Medusa, or EAGLE contract.

    Understand it in 90 seconds
    Problem
    Decoding K tokens from a large autoregressive Transformer takes K serial forward passes; each step is often limited by memory bandwidth rather than raw arithmetic throughput, leaving spare parallel capacity idle (Section 1).
    Core insight
    Speculative Decoding lets a small model Mq autoregressively draft gamma tokens, then runs the target model Mp in parallel on the prefix through each draft position to obtain distributions p1 through p{gamma+1}. Speculative sampling (rejection sampling plus an adjusted distribution) decides how many drafts to accept and adds one token guaranteed to come from Mp. The control point is lossless parallel verification versus step-by-step target-only decoding; the output distribution is identical to Mp alone (Algorithm 1, Appendix A.1).
    Strongest evidence
    T5-XXL 11B as Mp and off-the-shelf T5-small 77M as Mq versus the T5X baseline, batch=1, single TPU-v4 (Table 2): WMT EnDe 3.4X (temp=0, gamma=7, alpha=0.75) and 2.6X (temp=1, alpha=0.62); CNN/DM 3.1X and 2.3X. The abstract and Section 4 also report a 2X-3X band relative to T5X.
    Main boundary
    You need a task-aligned draft model and compute that can host gamma+1 parallel Mp forwards; total arithmetic operations can rise (Sections 3.4 and 6). This is a 2023 Google T5X experimental contract, not a vLLM or TensorRT-LLM product SLA, not GPTQ bitwidth, and not Medusa or EAGLE draft heads. InstructGPT 85±3% win rates, Transformer WMT BLEU, and YOLO mAP are not in this PDF.
    Read the full deep dive

  • RAG: Attach Retrieval to Generation, but Do Not Treat 2020 RAG as a Production RAG Platform

    Intermediate Retrieval, memory, and production RAG
    RAG deep reading: Part 1 , Note: Aug 27, 2026 , Paper: 2020 , NLP

    A source-grounded reading of Lewis et al., NeurIPS 2020: BART is paired with a dense retriever over Wikipedia, and RAG-Sequence / RAG-Token condition generation on retrieved passages. RAG-Seq reaches 44.5 Exact Match on NQ; this is a 2020 method paper, not a 2025 production RAG platform and not an agent loop.

    Understand it in 90 seconds
    Problem
    Large pretrained models store facts in their parameters, yet still lag task-specific architectures on knowledge-intensive work; parametric memory is hard to update, hard to inspect, and prone to hallucination.
    Core insight
    Pair a pretrained seq2seq generator (BART) with a pretrained dense retriever (initialized from DPR) over a Wikipedia index. The decision point moves from “answer from parameters only” to “retrieve passages, then condition generation.” RAG-Sequence shares one document across the sequence; RAG-Token may switch documents per token.
    Strongest evidence
    Table 1 open-domain QA: on NQ, RAG-Seq 44.5 and RAG-Token 44.1 beat DPR 41.5, REALM 40.4, and T5-11B+SSM 36.6. Table 2 generation and classification: on Open MS-MARCO, RAG-Seq is +2.6 Bleu and +2.6 Rouge-L versus BART; on FEVER-3, 72.5 sits 4.3 points below the then pipeline SOTA of 76.8, with no intermediate retrieval supervision.
    Main boundary
    The memory is the December 2018 Wikipedia dump split into 21M 100-word chunks, not a private corpus; retrieval is dense MIPS, not a production hybrid; there is no agentic search / read / final loop, and no 2026 enterprise sense of citation faithfulness.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact