Attention is All you Need101 Klaim Bersumber, Fakta yang Diperdebatkan & Bukti

Topic summary

The 2017 transformer paper: what it claimed, what it changed, and the disputes around credit.

Key research questions

• Organize claims about claims into a claim tree.

Important claims and supporting citations

• The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8.

  • https://arxiv.org/abs/1706.03762
  • https://arxiv.org/abs/1706.03762v1

• The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.

  • https://arxiv.org/pdf/1706.03762v1
  • https://arxiv.org/pdf/1706.03762v2
  • https://arxiv.org/pdf/1706.03762v3

• The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures.

  • https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  • https://arxiv.org/abs/1706.03762
  • https://arxiv.org/abs/1706.03762v1

• The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy.

  • https://arxiv.org/abs/1706.03762
  • https://arxiv.org/abs/1706.03762v1
  • https://papers.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html

• The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU.

  • https://arxiv.org/abs/1706.03762
  • https://arxiv.org/abs/1706.03762v1
  • https://papers.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html

• The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.

  • https://papers.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html

• Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0.

  • https://papers.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html
  • https://arxiv.org/abs/1706.03762
  • https://arxiv.org/abs/1706.03762v1

• An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential.

  • https://papers.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html

Competing explanations

• The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.

• The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution.

• The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention.

• The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer.

• The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor.

• The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer.

• In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness.

Outstanding questions

• Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper.
• Aidan Gomez was an intern from the University of Toronto when the paper was published.
• The path length between distant positions in RNNs grew as O(n).
• Niki Parmar is credited with designing, implementing, and evaluating model variants.
• Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase.
• ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model.
• Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models.
• Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation.