View all claims (text)
Claims
Claim text is for reading and citation. Bulk copying and AI training need a licensed maps data API key. Markdown
Attention Is All You Need paper details
Attention Is All You Need paper details
Labels: Inferred, Low
Prior work on attention in NMT
Prior work on attention in NMT
Labels: Inferred, Low
Preceding convolutional sequence models
Preceding convolutional sequence models
Labels: Inferred, Low
Arguments for Transformer architecture
Arguments for Transformer architecture
Labels: Inferred, Low
claim_104b5d55a4d80dddbeb75b47
The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples.
Labels: Inferred, Low, investigate
claim_25d22bc9b022d66e6d9fb46b
The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California.
Labels: Inferred, Low, investigate
claim_4335e804701eb2d27e59d83c
The "Attention Is All You Need" paper had 8 authors.
Labels: Inferred, Low, known
claim_45f72b45492f4c5c69385fcb
The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures.
Labels: Source-linked, High
claim_4a6f33f21d8d28e30780bdf8
The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023.
Labels: Source-linked, High
claim_50a7a769868fc4f59a99cbba
Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working.
Labels: Inferred, Low, investigate
claim_78c4d03acfda5c1e431b4342
The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, with individual contributions listed.
Labels: Inferred, Low, investigate
claim_a5dd3020d70fe88e48fcc606
Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate
claim_aad33d84b1d247813216ddc3
Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones.
Labels: Source-linked, High
claim_b80e9ab8420d41de82379488
An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential.
Labels: Source-linked, High
claim_0d47febb0e537915283262de
Noam Shazeer is credited with creating scaled dot-product attention mechanisms.
Labels: Inferred, Low
claim_10507064ebfbd9415ef97676
The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8.
Labels: Source-linked, High
claim_139d1b7fea36382e7b721bef
The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs).
Labels: Inferred, Low
claim_185da04e4ac0d53c5b99082a
The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing.
Labels: Inferred, Low
claim_2421b88eeac3865c33c09b83
The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.
Labels: Inferred, Low, known
claim_25af15c7ff66517d6c26ab05
The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models.
Labels: Inferred, Low
claim_25fd22118b2a09e17b031857
Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks.
Labels: Inferred, Low
claim_3a4651bba13bb101f5eeda23
The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12, 2017.
Labels: Source-linked, Low
claim_3a7d50c24ff46a8d85088538
The "Attention Is All You Need" paper introduced the Transformer architecture.
Labels: Inferred, Low
claim_41a4844534fd7c4b90870bcb
The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) conference.
Labels: Source-linked, Low
claim_46a5d61c940bc61b5a396996
The path length between distant positions in RNNs grew as O(n).
Labels: Inferred, Low, investigate
claim_48881a625dea2017f74a9c70
The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS.
Labels: Source-linked, Low
claim_4a36eddcc067c80e84a38897
The Transformer architecture allowed for more efficient computation on modern hardware.
Labels: Inferred, Low
claim_4d76cb4d9c8147188efeeaa8
The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualifying narratives that attribute results primarily to the attention architecture.
Labels: Inferred, High
claim_519d256a1fb1ad0a5daa1fbf
Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of encoder–decoder RNNs.
Labels: Inferred, Low, investigate
claim_51a8f842952cc360ff383900
Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural machine translation systems.
Labels: Inferred, Low
claim_527d67cee884c4d3c3a3f84d
Convolutional sequence models already had the same intra-example sequential-operation count as the models discussed in the context of the "Attention Is All You Need" paper.
Labels: Inferred, Low
claim_53c446607316ffd6cf7d6647
The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Long Beach, California.
Labels: Source-linked, Low
claim_5f4852fb0a40531df43ef49a
The author order in the "Attention Is All You Need" paper is noted to be random.
Labels: Inferred, Low
claim_7838b53c87fbbfb557a525cb
The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015.
Labels: Source-linked, Low
claim_8671fb9b2f5425721bd07edf
The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU.
Labels: Source-linked, High
claim_927a2631eb7f38c8ef98a855
Individual contributions of authors are specified in the "Attention Is All You Need" paper.
Labels: Inferred, Low
claim_955e920110f242d05c55c30d
By 2016–2017, attention was a standard component in competitive translation systems, always used in conjunction with recurrence.
Labels: Inferred, Low, investigate
claim_99035a6869c842fbf8099f84
ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model.
Labels: Inferred, Low, investigate
claim_9debcc10bf59d8bd5da6fdc3
The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.
Labels: Source-linked, High
claim_a8d8ea70456a26d0592b06fa
ByteNet, by Kalchbrenner et al. in 2016, used dilated convolutions and had logarithmic path length.
Labels: Inferred, Low, investigate
claim_b9a83bf140d0eddeb258e8ee
The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from existing methods.
Labels: Inferred, Low
claim_bfedc33e72975b4a27fa4b0a
The Transformer architecture eliminated the need for recurrence.
Labels: Inferred, Low
claim_c15150c72de3031520b6f3d8
Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0.
Labels: Source-linked, High
claim_cca93f2326999722b1467da0
The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural premise rather than a measured experiment.
Labels: Inferred, Low
claim_ccd26dae58615e121a02e0eb
The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples, tracks the paper's own Introduction almost word-for-word.
Labels: Inferred, Low
claim_cf7b5744004331163fcf1803
The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014.
Labels: Source-linked, Low
claim_d57e202f2dc33ba69f8da2ec
The "Attention Is All You Need" paper introduced the concept of self-attention.
Labels: Inferred, Low
claim_d712b01269c19f2eb9b11095
The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of three co-equal arguments used by the authors of "Attention Is All You Need."
Labels: Inferred, Low
claim_d784640207e1bc9ebb338b3f
Noam Shazeer is credited with creating multi-head attention mechanisms.
Labels: Inferred, Low
claim_d92dbf535e9d17e84f12dbd7
The latest official version of "Attention is All you Need" on arXiv is v7, not v5.
Labels: Source-linked, High
claim_dafa5deb89d2537da9ecc9dc
Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model.
Labels: Inferred, Low
claim_dd8d5477211f558e777b61e2
Luong et al. refined attention with multiplicative/dot-product attention and local attention in 2015.
Labels: Inferred, Low, investigate
claim_e5cfcd8c8338eb60c1c582d7
A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperparameters, had to be extracted from the code rather than the submitted paper.
Labels: Source-linked, High
claim_f79c6824562cf7ef914326ce
The Transformer model enables faster and more effective training through parallelization.
Labels: Inferred, Low
claim_0d2c97b07d6f43a44fc37581
Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate
claim_109b2b65de9358845145e539
Aidan Gomez was an intern from the University of Toronto when the paper was published.
Labels: Inferred, Low, investigate
claim_3b462da67628dea1b18ad066
Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research.
Labels: Inferred, Low, investigate
claim_5dbc904100ca44aa24c53b26
Niki Parmar is credited with designing, implementing, and evaluating model variants.
Labels: Inferred, Low, investigate
claim_6d0097192c0a222447405477
Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase.
Labels: Inferred, Low, investigate
claim_812a02b718b83b27da1a8639
The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy.
Labels: Source-linked, High
claim_9ca1153e396783334d865203
Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models.
Labels: Inferred, Low, investigate
claim_9d4b882fbeaed19302ec9237
Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation.
Labels: Inferred, Low, investigate
claim_b102d8f4e92a851157df7923
Llion Jones is credited with visualizations and efficient inference work.
Labels: Inferred, Low, investigate
claim_da3643812bb1e6f193eedc49
The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revision, not in the original posting.
Labels: Inferred, High
claim_094f0cc9e843ff87fb2b7137
Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the University of Toronto.
Labels: Inferred, Low
claim_0aa591ea5d4c6243a596698e
The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor.
Labels: Contested, High
claim_0b98ca1e8cf109d0371bd8f5
Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszkaiser@google.com.
Labels: Source-linked, Medium
claim_0e4a5e8e5d71048234414eb6
Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You Need" paper.
Labels: Inferred, Low
claim_10f7df62feefb096e46c5130
Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2016 discussion.
Labels: Inferred, High
claim_139d1d390cfb472dbbe1f818
Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain.
Labels: Inferred, Low
claim_1582628a796f38cb1aa6f44b
The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.
Labels: Contested, High
claim_1c8e5efb564db325461219ed
Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires.
Labels: Inferred, Medium
claim_2351ca4124f8b5ca525d28b4
Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need".
Labels: Inferred, Low
claim_2905404d3f41f7914360eace
The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research organization structure around 2016–2017.
Labels: Inferred, Medium
claim_29b7f293e56db098c430cfd9
Aidan Gomez's post-paper trajectory corroborates the "intern" framing.
Labels: Source-linked, Low
claim_2ce24a09545c8a197dc2e75d
Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, shared with Kaiser.
Labels: Inferred, Low
claim_2ef74462624f17a1105b7885
After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher.
Labels: Source-linked, Low
claim_3b0c2b5650bede0a8d5b9793
Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017.
Labels: Inferred, Low
claim_40a8b51324a07ffac3abd1fd
Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google.
Labels: Inferred, Medium
claim_4b10c80ed716e7b07f329150
The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer.
Labels: Contested, High
claim_5360e74850a613b537dcdfa9
Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@google.com.
Labels: Source-linked, Medium
claim_627f8468d869594aad311db2
Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain.
Labels: Source-linked, Low
claim_6c2f25f51fdc506e74eae9ef
Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswani@google.com.
Labels: Source-linked, Medium
claim_6f7aaf6af412bef85f08099c
Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain.
Labels: Inferred, Medium
claim_7329262d557cb3c88da6502d
Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appeared in June 2017.
Labels: Source-linked, Low
claim_7e21aad5fc8968b329c9459f
In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness.
Labels: Contested, High
claim_82da7a91cd09f744e66bf490
Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advance human understanding.”
Labels: Inferred, Medium
claim_879b1092306147a9b6c53f69
Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a university rather than Google Brain or Google Research.
Labels: Source-linked, Low
claim_9bcb2a40fd7529f5d29441cb
The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention.
Labels: Contested, High
claim_9c9e077cd131dcbb7d81303a
Aidan Gomez was mentored by Łukasz Kaiser at Google Brain.
Labels: Inferred, Low
claim_a5d9d87e695550c4173e7e6b
Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work performed while at Google Research.”
Labels: Source-linked, Medium
claim_a7d566a2f6a4788226bbb2cb
Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage internship task.
Labels: Inferred, Low
claim_c2bd06aeb23fc5e699a37031
Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natural-language inference and was trivially parallelizable.
Labels: Inferred, High
claim_cebce0e0fd2071c866cfca51
Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receiving credit.
Labels: Inferred, High
claim_d3152341d9c9e78ad12f1863
The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran.
Labels: Inferred, Low
claim_d68ea69fb5a7c1512fcd27d7
Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer project crystallized in early 2017.
Labels: Inferred, Low
claim_d974d47be68e901004343515
Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA team) on the initial Transformer designs.
Labels: Inferred, Medium
claim_e55cf2199bd87da2d9705d39
Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attention Is All You Need" paper.
Labels: Source-linked, Medium
claim_ef7f401d9060e217a4d7b656
The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution.
Labels: Contested, High
claim_fe133f878247b5aa365b2a27
Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July.
Labels: Inferred, Medium
claim_ff061eb3d5f7c5f5753f0034
The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer.
Labels: Contested, High