# Attention is All you Need

Authors: nineDan
HTML: https://moyomap.com/public-maps/kQ84228NBSVSr1P0aEwE
Markdown: https://moyomap.com/public-maps/kQ84228NBSVSr1P0aEwE.md
License: HTML and markdown are for reading and citation. Bulk copying and AI training need a written license. See /terms and /developers#maps-data.

### Attention Is All You Need paper details

Attention Is All You Need paper details
Labels: Inferred, Low

### Prior work on attention in NMT

Prior work on attention in NMT
Labels: Inferred, Low

### Preceding convolutional sequence models

Preceding convolutional sequence models
Labels: Inferred, Low

### Arguments for Transformer architecture

Arguments for Transformer architecture
Labels: Inferred, Low

#### claim_104b5d55a4d80dddbeb75b47

The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples.
Labels: Inferred, Low, investigate

#### claim_25d22bc9b022d66e6d9fb46b

The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California.
Labels: Inferred, Low, investigate

#### claim_4335e804701eb2d27e59d83c

The "Attention Is All You Need" paper had 8 authors.
Labels: Inferred, Low, known

#### claim_45f72b45492f4c5c69385fcb

The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures.
Labels: Source-linked, High

#### claim_4a6f33f21d8d28e30780bdf8

The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023.
Labels: Source-linked, High

#### claim_50a7a769868fc4f59a99cbba

Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working.
Labels: Inferred, Low, investigate

#### claim_78c4d03acfda5c1e431b4342

The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, with individual contributions listed.
Labels: Inferred, Low, investigate

#### claim_a5dd3020d70fe88e48fcc606

Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate

#### claim_aad33d84b1d247813216ddc3

Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones.
Labels: Source-linked, High

#### claim_b80e9ab8420d41de82379488

An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential.
Labels: Source-linked, High

##### claim_0d47febb0e537915283262de

Noam Shazeer is credited with creating scaled dot-product attention mechanisms.
Labels: Inferred, Low

##### claim_10507064ebfbd9415ef97676

The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8.
Labels: Source-linked, High

##### claim_139d1b7fea36382e7b721bef

The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs).
Labels: Inferred, Low

##### claim_185da04e4ac0d53c5b99082a

The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing.
Labels: Inferred, Low

##### claim_2421b88eeac3865c33c09b83

The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.
Labels: Inferred, Low, known

##### claim_25af15c7ff66517d6c26ab05

The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models.
Labels: Inferred, Low

##### claim_25fd22118b2a09e17b031857

Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks.
Labels: Inferred, Low

##### claim_3a4651bba13bb101f5eeda23

The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12, 2017.
Labels: Source-linked, Low

##### claim_3a7d50c24ff46a8d85088538

The "Attention Is All You Need" paper introduced the Transformer architecture.
Labels: Inferred, Low

##### claim_41a4844534fd7c4b90870bcb

The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) conference.
Labels: Source-linked, Low

##### claim_46a5d61c940bc61b5a396996

The path length between distant positions in RNNs grew as O(n).
Labels: Inferred, Low, investigate

##### claim_48881a625dea2017f74a9c70

The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS.
Labels: Source-linked, Low

##### claim_4a36eddcc067c80e84a38897

The Transformer architecture allowed for more efficient computation on modern hardware.
Labels: Inferred, Low

##### claim_4d76cb4d9c8147188efeeaa8

The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualifying narratives that attribute results primarily to the attention architecture.
Labels: Inferred, High

##### claim_519d256a1fb1ad0a5daa1fbf

Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of encoder–decoder RNNs.
Labels: Inferred, Low, investigate

##### claim_51a8f842952cc360ff383900

Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural machine translation systems.
Labels: Inferred, Low

##### claim_527d67cee884c4d3c3a3f84d

Convolutional sequence models already had the same intra-example sequential-operation count as the models discussed in the context of the "Attention Is All You Need" paper.
Labels: Inferred, Low

##### claim_53c446607316ffd6cf7d6647

The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Long Beach, California.
Labels: Source-linked, Low

##### claim_5f4852fb0a40531df43ef49a

The author order in the "Attention Is All You Need" paper is noted to be random.
Labels: Inferred, Low

##### claim_7838b53c87fbbfb557a525cb

The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015.
Labels: Source-linked, Low

##### claim_8671fb9b2f5425721bd07edf

The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU.
Labels: Source-linked, High

##### claim_927a2631eb7f38c8ef98a855

Individual contributions of authors are specified in the "Attention Is All You Need" paper.
Labels: Inferred, Low

##### claim_955e920110f242d05c55c30d

By 2016–2017, attention was a standard component in competitive translation systems, always used in conjunction with recurrence.
Labels: Inferred, Low, investigate

##### claim_99035a6869c842fbf8099f84

ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model.
Labels: Inferred, Low, investigate

##### claim_9debcc10bf59d8bd5da6fdc3

The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.
Labels: Source-linked, High

##### claim_a8d8ea70456a26d0592b06fa

ByteNet, by Kalchbrenner et al. in 2016, used dilated convolutions and had logarithmic path length.
Labels: Inferred, Low, investigate

##### claim_b9a83bf140d0eddeb258e8ee

The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from existing methods.
Labels: Inferred, Low

##### claim_bfedc33e72975b4a27fa4b0a

The Transformer architecture eliminated the need for recurrence.
Labels: Inferred, Low

##### claim_c15150c72de3031520b6f3d8

Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0.
Labels: Source-linked, High

##### claim_cca93f2326999722b1467da0

The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural premise rather than a measured experiment.
Labels: Inferred, Low

##### claim_ccd26dae58615e121a02e0eb

The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples, tracks the paper's own Introduction almost word-for-word.
Labels: Inferred, Low

##### claim_cf7b5744004331163fcf1803

The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014.
Labels: Source-linked, Low

##### claim_d57e202f2dc33ba69f8da2ec

The "Attention Is All You Need" paper introduced the concept of self-attention.
Labels: Inferred, Low

##### claim_d712b01269c19f2eb9b11095

The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of three co-equal arguments used by the authors of "Attention Is All You Need."
Labels: Inferred, Low

##### claim_d784640207e1bc9ebb338b3f

Noam Shazeer is credited with creating multi-head attention mechanisms.
Labels: Inferred, Low

##### claim_d92dbf535e9d17e84f12dbd7

The latest official version of "Attention is All you Need" on arXiv is v7, not v5.
Labels: Source-linked, High

##### claim_dafa5deb89d2537da9ecc9dc

Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model.
Labels: Inferred, Low

##### claim_dd8d5477211f558e777b61e2

Luong et al. refined attention with multiplicative/dot-product attention and local attention in 2015.
Labels: Inferred, Low, investigate

##### claim_e5cfcd8c8338eb60c1c582d7

A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperparameters, had to be extracted from the code rather than the submitted paper.
Labels: Source-linked, High

##### claim_f79c6824562cf7ef914326ce

The Transformer model enables faster and more effective training through parallelization.
Labels: Inferred, Low

###### claim_0d2c97b07d6f43a44fc37581

Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate

###### claim_109b2b65de9358845145e539

Aidan Gomez was an intern from the University of Toronto when the paper was published.
Labels: Inferred, Low, investigate

###### claim_3b462da67628dea1b18ad066

Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research.
Labels: Inferred, Low, investigate

###### claim_5dbc904100ca44aa24c53b26

Niki Parmar is credited with designing, implementing, and evaluating model variants.
Labels: Inferred, Low, investigate

###### claim_6d0097192c0a222447405477

Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase.
Labels: Inferred, Low, investigate

###### claim_812a02b718b83b27da1a8639

The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy.
Labels: Source-linked, High

###### claim_9ca1153e396783334d865203

Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models.
Labels: Inferred, Low, investigate

###### claim_9d4b882fbeaed19302ec9237

Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation.
Labels: Inferred, Low, investigate

###### claim_b102d8f4e92a851157df7923

Llion Jones is credited with visualizations and efficient inference work.
Labels: Inferred, Low, investigate

###### claim_da3643812bb1e6f193eedc49

The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revision, not in the original posting.
Labels: Inferred, High

###### claim_094f0cc9e843ff87fb2b7137

Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the University of Toronto.
Labels: Inferred, Low

###### claim_0aa591ea5d4c6243a596698e

The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor.
Labels: Contested, High

###### claim_0b98ca1e8cf109d0371bd8f5

Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszkaiser@google.com.
Labels: Source-linked, Medium

###### claim_0e4a5e8e5d71048234414eb6

Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You Need" paper.
Labels: Inferred, Low

###### claim_10f7df62feefb096e46c5130

Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2016 discussion.
Labels: Inferred, High

###### claim_139d1d390cfb472dbbe1f818

Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain.
Labels: Inferred, Low

###### claim_1582628a796f38cb1aa6f44b

The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.
Labels: Contested, High

###### claim_1c8e5efb564db325461219ed

Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires.
Labels: Inferred, Medium

###### claim_2351ca4124f8b5ca525d28b4

Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need".
Labels: Inferred, Low

###### claim_2905404d3f41f7914360eace

The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research organization structure around 2016–2017.
Labels: Inferred, Medium

###### claim_29b7f293e56db098c430cfd9

Aidan Gomez's post-paper trajectory corroborates the "intern" framing.
Labels: Source-linked, Low

###### claim_2ce24a09545c8a197dc2e75d

Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, shared with Kaiser.
Labels: Inferred, Low

###### claim_2ef74462624f17a1105b7885

After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher.
Labels: Source-linked, Low

###### claim_3b0c2b5650bede0a8d5b9793

Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017.
Labels: Inferred, Low

###### claim_40a8b51324a07ffac3abd1fd

Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google.
Labels: Inferred, Medium

###### claim_4b10c80ed716e7b07f329150

The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer.
Labels: Contested, High

###### claim_5360e74850a613b537dcdfa9

Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@google.com.
Labels: Source-linked, Medium

###### claim_627f8468d869594aad311db2

Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain.
Labels: Source-linked, Low

###### claim_6c2f25f51fdc506e74eae9ef

Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswani@google.com.
Labels: Source-linked, Medium

###### claim_6f7aaf6af412bef85f08099c

Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain.
Labels: Inferred, Medium

###### claim_7329262d557cb3c88da6502d

Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appeared in June 2017.
Labels: Source-linked, Low

###### claim_7e21aad5fc8968b329c9459f

In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness.
Labels: Contested, High

###### claim_82da7a91cd09f744e66bf490

Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advance human understanding.”
Labels: Inferred, Medium

###### claim_879b1092306147a9b6c53f69

Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a university rather than Google Brain or Google Research.
Labels: Source-linked, Low

###### claim_9bcb2a40fd7529f5d29441cb

The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention.
Labels: Contested, High

###### claim_9c9e077cd131dcbb7d81303a

Aidan Gomez was mentored by Łukasz Kaiser at Google Brain.
Labels: Inferred, Low

###### claim_a5d9d87e695550c4173e7e6b

Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work performed while at Google Research.”
Labels: Source-linked, Medium

###### claim_a7d566a2f6a4788226bbb2cb

Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage internship task.
Labels: Inferred, Low

###### claim_c2bd06aeb23fc5e699a37031

Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natural-language inference and was trivially parallelizable.
Labels: Inferred, High

###### claim_cebce0e0fd2071c866cfca51

Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receiving credit.
Labels: Inferred, High

###### claim_d3152341d9c9e78ad12f1863

The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran.
Labels: Inferred, Low

###### claim_d68ea69fb5a7c1512fcd27d7

Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer project crystallized in early 2017.
Labels: Inferred, Low

###### claim_d974d47be68e901004343515

Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA team) on the initial Transformer designs.
Labels: Inferred, Medium

###### claim_e55cf2199bd87da2d9705d39

Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attention Is All You Need" paper.
Labels: Source-linked, Medium

###### claim_ef7f401d9060e217a4d7b656

The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution.
Labels: Contested, High

###### claim_fe133f878247b5aa365b2a27

Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July.
Labels: Inferred, Medium

###### claim_ff061eb3d5f7c5f5753f0034

The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer.
Labels: Contested, High