Original Transformer deep reading

Series · 1 posts

← Back to Paper Reading

  • Transformer: Drop Recurrence for Self-Attention, but WMT 2017 BLEU Does Not Represent Later LLMs

    Intermediate Build the foundations first
    Original Transformer deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2017 , NLP

    A source-grounded reading of Vaswani et al., NeurIPS 2017 / arXiv:1706.03762: stacked encoder-decoder with multi-head self-attention and positional encodings replaces RNNs and convolutions for machine translation. On WMT 2014, the big model reaches 28.4 BLEU EN-DE and 41.8 BLEU EN-FR; this is 2017 sequence-transduction evidence, not a BERT, GPT-3, or ViT contract.

    Understand it in 90 seconds
    Problem
    Before 2017, state-of-the-art sequence transduction (seq2seq) relied on RNN/LSTM/GRU encoder-decoders whose computation unfolds along time steps, limiting within-example parallelism on long sequences. Bahdanau et al. already attached attention to RNNs, but recurrence remained the backbone (Sections 1-2).
    Core insight
    The Transformer stacks multi-head self-attention and position-wise FFN in both encoder and decoder, injects order with sinusoidal positional encodings, and removes recurrence and convolutions entirely. The control point is parallel global attention versus sequential hidden states; path length for long-range dependencies is $O(1)$ per layer (Table 1).
    Strongest evidence
    WMT 2014 newstest2014 (Table 2): Transformer (big) reaches 28.4 BLEU EN-DE (above prior bests including ensembles) and 41.8 BLEU EN-FR; big-model training takes 3.5 days on 8xP100 GPUs (300K steps). The base model reaches 27.3 BLEU EN-DE with training FLOPs of $3.3\times10^{18}$, below GNMT+RL at $2.3\times10^{19}$. Hardware text: base training 12 hours / 100K steps at 0.4 seconds per step (Section 5.2).
    Main boundary
    The task is supervised MT encoder-decoder, not a pretrained language model, not bidirectional BERT, not decoder-only GPT, and not ViT. BERT, GPT-2/3, T5, LLaMA, and ChatGPT benchmarks are not in this PDF; YOLO VOC mAP and ResNet ImageNet 4.49% are not MT contracts either.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact