# Did the Attention Mechanism Disguise the True Role of Feed-Forward Storage?: 122 Sourced Claims, Contested Facts & Evidence Far from attention being all that was needed, feed-forward layers perform the vast majority of factual computation and parameter storage. Authors: nineDan HTML: https://moyomap.com/maps/did-attention-mechanism-disguise Markdown: https://moyomap.com/maps/did-attention-mechanism-disguise.md License: HTML and markdown are for reading and citation. Bulk copying and AI training need a written license. See /terms and /developers#maps-data. Cite a claim as: MoyoMap, "Did the Attention Mechanism Disguise the True Role of Feed-Forward Storage?: 122 Sourced Claims, Contested Facts & Evidence", claim {id}, https://moyomap.com/maps/did-attention-mechanism-disguise.md#{id}. “Moyo, a product of SenTeGuard Inc.” ### The 'Attention' Misnomer and MLP Dominance Id: claim_cb5921314435999fec5c3ae5 Parent: topic_tEM4AfuQeisWgQUj3Lb5 Depth: 1 Feed-forward sublayers function as associative key-value memories that store the substantive knowledge in Transformer models. Labels: Inferred, High, investigate ### Attention Is All You Need paper details Id: section_3bb0f6ddaccd61d270a97258 Parent: topic_tEM4AfuQeisWgQUj3Lb5 Depth: 1 Attention Is All You Need paper details Labels: Inferred, Low ### Prior work on attention in NMT Id: section_678cb7f2e9d079a0f7423515 Parent: topic_tEM4AfuQeisWgQUj3Lb5 Depth: 1 Prior work on attention in NMT Labels: Inferred, Low ### Preceding convolutional sequence models Id: section_e3f89c305de932b096e7d6f5 Parent: topic_tEM4AfuQeisWgQUj3Lb5 Depth: 1 Preceding convolutional sequence models Labels: Inferred, Low ### Arguments for Transformer architecture Id: section_e63c905fa4d2338e5c698455 Parent: topic_tEM4AfuQeisWgQUj3Lb5 Depth: 1 Arguments for Transformer architecture Labels: Inferred, Low #### ROME's key-value memory formulation relies on classical linear associative memory frameworks originally estab… Id: claim_0df8bece754dcd6d57bb2710 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 ROME's key-value memory formulation relies on classical linear associative memory frameworks originally established by Anderson (1972) and Kohonen (1972). Labels: Inferred, Low #### The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs preven… Id: claim_104b5d55a4d80dddbeb75b47 Parent: section_e63c905fa4d2338e5c698455 Depth: 2 The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples. Labels: Inferred, Low, investigate #### Under ROME's mechanistic model, summed subject information accumulated by MLPs is copied to the last token by… Id: claim_206e668e04b14939066ed559 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Under ROME's mechanistic model, summed subject information accumulated by MLPs is copied to the last token by attention mechanisms at high layers. Labels: Inferred, Low #### The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NI… Id: claim_25d22bc9b022d66e6d9fb46b Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California. Labels: Inferred, Low, investigate #### Associative-memory localization and ROME model editing techniques transfer successfully to Mamba, a non-trans… Id: claim_3aa2e5f82ee31f4a51ee1b12 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Associative-memory localization and ROME model editing techniques transfer successfully to Mamba, a non-transformer state-space model. Labels: Inferred, Low #### The original implementation of Causal Tracing in ROME restores 10 layers simultaneously, indicating that caus… Id: claim_3fdcb89f015b303ed8c4122d Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 The original implementation of Causal Tracing in ROME restores 10 layers simultaneously, indicating that causal signals are spread across multiple consecutive MLPs rather than isolated to a single layer. Labels: Inferred, Low #### The "Attention Is All You Need" paper had 8 authors. Id: claim_4335e804701eb2d27e59d83c Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper had 8 authors. Labels: Inferred, Low, known #### The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent Engli… Id: claim_45f72b45492f4c5c69385fcb Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures. Labels: Source-linked, High #### The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2… Id: claim_4a6f33f21d8d28e30780bdf8 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023. Labels: Source-linked, High #### Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already… Id: claim_50a7a769868fc4f59a99cbba Parent: section_e3f89c305de932b096e7d6f5 Depth: 2 Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working. Labels: Inferred, Low, investigate #### Geva et al. Id: claim_531397192911a79fa879fecf Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Geva et al. (2023) show that subject representations are enriched with attributes by early MLP sublayers, while attribute extraction is carried out by attention heads that encode subject-attribute mappings in their parameters. Labels: Inferred, Low #### Hase et al. Id: claim_59992daee46ff30a9c1bf5e4 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Hase et al. found that representation zeroing yields near-zero correlations with edit success across layers, matching the non-predictiveness found with representation denoising. Labels: Inferred, Low #### In model editing experiments by Hase et al., layer choice accounts for 94.7% of the variance in rewrite score… Id: claim_5d20e9b7b3671859d7586fcf Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 In model editing experiments by Hase et al., layer choice accounts for 94.7% of the variance in rewrite score, while adding causal tracing effects increases R² by only 0.1% to 94.8%. Labels: Inferred, Low #### Specific attention heads directly encode subject-attribute mappings in 30.2% of cases in GPT-2 and 39.3% in G… Id: claim_603cdd1863e58f0308512103 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Specific attention heads directly encode subject-attribute mappings in 30.2% of cases in GPT-2 and 39.3% in GPT-J, with extraction behavior observed in roughly 70% of predictions. Labels: Inferred, Low #### The key-value associative memory framing of transformer FFNs draws directly on Sukhbaatar et al. Id: claim_6c418964c893035c712c5544 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 The key-value associative memory framing of transformer FFNs draws directly on Sukhbaatar et al. (2015) End-to-End Memory Networks. Labels: Inferred, Low #### The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, w… Id: claim_78c4d03acfda5c1e431b4342 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, with individual contributions listed. Labels: Inferred, Low, investigate #### Causal intervention in ROME localizes factual association specifically to middle-layer feed-forward modules w… Id: claim_7f6695d22c75c6025a82ba0a Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Causal intervention in ROME localizes factual association specifically to middle-layer feed-forward modules while processing the subject's last token. Labels: Inferred, Low #### Hase et al. Id: claim_99052e5e5d0dca42da41b019 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Hase et al. (2023) demonstrated that causal localization via representation denoising (Causal Tracing) is statistically uncorrelated with the success of fact-injecting edits across MLP layers. Labels: Inferred, Low #### Lv et al. Id: claim_a32c2d64206fa0bbdc8a8241 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Lv et al. found that task-specific attention heads move topic entities to the final position while MLPs execute the relation function. Labels: Inferred, Low #### Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is A… Id: claim_a5dd3020d70fe88e48fcc606 Parent: section_678cb7f2e9d079a0f7423515 Depth: 2 Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is All You Need" paper. Labels: Inferred, Low, investigate #### Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the curr… Id: claim_aad33d84b1d247813216ddc3 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones. Labels: Source-linked, High #### An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and foun… Id: claim_b80e9ab8420d41de82379488 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential. Labels: Source-linked, High #### Knocking out early MLP sublayers in autoregressive models leads to up to an approximately 88% reduction in th… Id: claim_b9f0173dbbddc173bc770754 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Knocking out early MLP sublayers in autoregressive models leads to up to an approximately 88% reduction in the attributes rate of subject representations. Labels: Inferred, Low #### In the Fact Forcing variant tested by Hase et al., causal tracing results correlate with edit success, though… Id: claim_bfacfaf8e6e0ceeb811c275e Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 In the Fact Forcing variant tested by Hase et al., causal tracing results correlate with edit success, though selecting early-to-mid-layer MLP weights without tracing remains superior. Labels: Inferred, Low #### In transformer feed-forward layers, shallow layers only detect shallow sentence patterns, whereas upper layer… Id: claim_c47a7f8148df1c9ba1ce2b98 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 In transformer feed-forward layers, shallow layers only detect shallow sentence patterns, whereas upper layers encode semantic content. Labels: Inferred, Low #### Chen et al. Id: claim_d1e35a93f58e41c958e6b8c9 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Chen et al. found that multiple distinct sets of knowledge neurons can store identical facts rather than facts being isolated to a single MLP location. Labels: Inferred, Low #### Geva et al. Id: claim_df89045c2d494ba82c72de9c Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Geva et al. (2021) state that feed-forward layers constitute two-thirds of a transformer model's parameters and operate as key-value memories where keys match training-corpus textual patterns and values induce output-vocabulary distributions. Labels: Inferred, Low #### Feed-forward layer outputs in transformers are compositions of multiple memories refined across layers rather… Id: claim_f59ded57f6fe26d46b1a315a Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Feed-forward layer outputs in transformers are compositions of multiple memories refined across layers rather than single-memory lookups. Labels: Inferred, Low #### Associations edited by ROME are directional, meaning reverse relational queries (e.g., subject-object inversi… Id: claim_f73531de61d22de1d948a1f3 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Associations edited by ROME are directional, meaning reverse relational queries (e.g., subject-object inversion) require independent edits. Labels: Inferred, Low #### Pre-edit facts survive inside later-layer attention heads within specific left singular vectors of the attent… Id: claim_fcd6d40300cdaa5d631c9437 Parent: claim_cb5921314435999fec5c3ae5 Depth: 2 Pre-edit facts survive inside later-layer attention heads within specific left singular vectors of the attention output matrix after MLP editing (including ROME, MEMIT, and AlphaEdit) and can be recovered by ablating those vectors. Labels: Inferred, Low ##### Noam Shazeer is credited with creating scaled dot-product attention mechanisms. Id: claim_0d47febb0e537915283262de Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Noam Shazeer is credited with creating scaled dot-product attention mechanisms. Labels: Inferred, Low ##### The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of… Id: claim_10507064ebfbd9415ef97676 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8. Labels: Source-linked, High ##### The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs). Id: claim_139d1b7fea36382e7b721bef Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs). Labels: Inferred, Low ##### The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing. Id: claim_185da04e4ac0d53c5b99082a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing. Labels: Inferred, Low ##### The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Us… Id: claim_2421b88eeac3865c33c09b83 Parent: claim_4335e804701eb2d27e59d83c Depth: 3 The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Labels: Inferred, Low, known ##### The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models. Id: claim_25af15c7ff66517d6c26ab05 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models. Labels: Inferred, Low ##### Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks. Id: claim_25fd22118b2a09e17b031857 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks. Labels: Inferred, Low ##### The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12,… Id: claim_3a4651bba13bb101f5eeda23 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12, 2017. Labels: Source-linked, Low ##### The "Attention Is All You Need" paper introduced the Transformer architecture. Id: claim_3a7d50c24ff46a8d85088538 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper introduced the Transformer architecture. Labels: Inferred, Low ##### The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) confe… Id: claim_41a4844534fd7c4b90870bcb Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) conference. Labels: Source-linked, Low ##### The path length between distant positions in RNNs grew as O(n). Id: claim_46a5d61c940bc61b5a396996 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The path length between distant positions in RNNs grew as O(n). Labels: Inferred, Low, investigate ##### The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS. Id: claim_48881a625dea2017f74a9c70 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS. Labels: Source-linked, Low ##### The Transformer architecture allowed for more efficient computation on modern hardware. Id: claim_4a36eddcc067c80e84a38897 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer architecture allowed for more efficient computation on modern hardware. Labels: Inferred, Low ##### The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualif… Id: claim_4d76cb4d9c8147188efeeaa8 Parent: claim_b80e9ab8420d41de82379488 Depth: 3 The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualifying narratives that attribute results primarily to the attention architecture. Labels: Inferred, High ##### Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of enc… Id: claim_519d256a1fb1ad0a5daa1fbf Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of encoder–decoder RNNs. Labels: Inferred, Low, investigate ##### Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural mach… Id: claim_51a8f842952cc360ff383900 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural machine translation systems. Labels: Inferred, Low ##### Convolutional sequence models already had the same intra-example sequential-operation count as the models dis… Id: claim_527d67cee884c4d3c3a3f84d Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 Convolutional sequence models already had the same intra-example sequential-operation count as the models discussed in the context of the "Attention Is All You Need" paper. Labels: Inferred, Low ##### The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Lo… Id: claim_53c446607316ffd6cf7d6647 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Long Beach, California. Labels: Source-linked, Low ##### The author order in the "Attention Is All You Need" paper is noted to be random. Id: claim_5f4852fb0a40531df43ef49a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The author order in the "Attention Is All You Need" paper is noted to be random. Labels: Inferred, Low ##### The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015. Id: claim_7838b53c87fbbfb557a525cb Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015. Labels: Source-linked, Low ##### The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scor… Id: claim_8671fb9b2f5425721bd07edf Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU. Labels: Source-linked, High ##### Individual contributions of authors are specified in the "Attention Is All You Need" paper. Id: claim_927a2631eb7f38c8ef98a855 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Individual contributions of authors are specified in the "Attention Is All You Need" paper. Labels: Inferred, Low ##### By 2016–2017, attention was a standard component in competitive translation systems, always used in conjuncti… Id: claim_955e920110f242d05c55c30d Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 By 2016–2017, attention was a standard component in competitive translation systems, always used in conjunction with recurrence. Labels: Inferred, Low, investigate ##### ConvS2S / Fairseq, by Gehring et al. Id: claim_99035a6869c842fbf8099f84 Parent: claim_50a7a769868fc4f59a99cbba Depth: 3 ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model. Labels: Inferred, Low, investigate ##### The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1. Id: claim_9debcc10bf59d8bd5da6fdc3 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1. Labels: Source-linked, High ##### ByteNet, by Kalchbrenner et al. Id: claim_a8d8ea70456a26d0592b06fa Parent: claim_50a7a769868fc4f59a99cbba Depth: 3 ByteNet, by Kalchbrenner et al. in 2016, used dilated convolutions and had logarithmic path length. Labels: Inferred, Low, investigate ##### The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from… Id: claim_b9a83bf140d0eddeb258e8ee Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from existing methods. Labels: Inferred, Low ##### The Transformer architecture eliminated the need for recurrence. Id: claim_bfedc33e72975b4a27fa4b0a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer architecture eliminated the need for recurrence. Labels: Inferred, Low ##### Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score o… Id: claim_c15150c72de3031520b6f3d8 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0. Labels: Source-linked, High ##### The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural… Id: claim_cca93f2326999722b1467da0 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural premise rather than a measured experiment. Labels: Inferred, Low ##### The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that… Id: claim_ccd26dae58615e121a02e0eb Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples, tracks the paper's own Introduction almost word-for-word. Labels: Inferred, Low ##### The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014. Id: claim_cf7b5744004331163fcf1803 Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014. Labels: Source-linked, Low ##### The "Attention Is All You Need" paper introduced the concept of self-attention. Id: claim_d57e202f2dc33ba69f8da2ec Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper introduced the concept of self-attention. Labels: Inferred, Low ##### The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of thre… Id: claim_d712b01269c19f2eb9b11095 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of three co-equal arguments used by the authors of "Attention Is All You Need." Labels: Inferred, Low ##### Noam Shazeer is credited with creating multi-head attention mechanisms. Id: claim_d784640207e1bc9ebb338b3f Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Noam Shazeer is credited with creating multi-head attention mechanisms. Labels: Inferred, Low ##### The latest official version of "Attention is All you Need" on arXiv is v7, not v5. Id: claim_d92dbf535e9d17e84f12dbd7 Parent: claim_4a6f33f21d8d28e30780bdf8 Depth: 3 The latest official version of "Attention is All you Need" on arXiv is v7, not v5. Labels: Source-linked, High ##### Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model. Id: claim_dafa5deb89d2537da9ecc9dc Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model. Labels: Inferred, Low ##### Luong et al. Id: claim_dd8d5477211f558e777b61e2 Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 Luong et al. refined attention with multiplicative/dot-product attention and local attention in 2015. Labels: Inferred, Low, investigate ##### A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperpar… Id: claim_e5cfcd8c8338eb60c1c582d7 Parent: claim_b80e9ab8420d41de82379488 Depth: 3 A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperparameters, had to be extracted from the code rather than the submitted paper. Labels: Source-linked, High ##### The Transformer model enables faster and more effective training through parallelization. Id: claim_f79c6824562cf7ef914326ce Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model enables faster and more effective training through parallelization. Labels: Inferred, Low ###### Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Ne… Id: claim_0d2c97b07d6f43a44fc37581 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper. Labels: Inferred, Low, investigate ###### Aidan Gomez was an intern from the University of Toronto when the paper was published. Id: claim_109b2b65de9358845145e539 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Aidan Gomez was an intern from the University of Toronto when the paper was published. Labels: Inferred, Low, investigate ###### Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research. Id: claim_3b462da67628dea1b18ad066 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research. Labels: Inferred, Low, investigate ###### Niki Parmar is credited with designing, implementing, and evaluating model variants. Id: claim_5dbc904100ca44aa24c53b26 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Niki Parmar is credited with designing, implementing, and evaluating model variants. Labels: Inferred, Low, investigate ###### Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase. Id: claim_6d0097192c0a222447405477 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase. Labels: Inferred, Low, investigate ###### The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/b… Id: claim_812a02b718b83b27da1a8639 Parent: claim_8671fb9b2f5425721bd07edf Depth: 4 The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy. Labels: Source-linked, High ###### Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models. Id: claim_9ca1153e396783334d865203 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models. Labels: Inferred, Low, investigate ###### Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free posi… Id: claim_9d4b882fbeaed19302ec9237 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation. Labels: Inferred, Low, investigate ###### Llion Jones is credited with visualizations and efficient inference work. Id: claim_b102d8f4e92a851157df7923 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Llion Jones is credited with visualizations and efficient inference work. Labels: Inferred, Low, investigate ###### The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revisi… Id: claim_da3643812bb1e6f193eedc49 Parent: claim_10507064ebfbd9415ef97676 Depth: 4 The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revision, not in the original posting. Labels: Inferred, High ###### Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the Universi… Id: claim_094f0cc9e843ff87fb2b7137 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the University of Toronto. Labels: Inferred, Low ###### The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole ar… Id: claim_0aa591ea5d4c6243a596698e Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor. Labels: Contested, High ###### Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszka… Id: claim_0b98ca1e8cf109d0371bd8f5 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszkaiser@google.com. Labels: Source-linked, Medium ###### Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You… Id: claim_0e4a5e8e5d71048234414eb6 Parent: claim_109b2b65de9358845145e539 Depth: 5 Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You Need" paper. Labels: Inferred, Low ###### Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2… Id: claim_10f7df62feefb096e46c5130 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2016 discussion. Labels: Inferred, High ###### Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain. Id: claim_139d1d390cfb472dbbe1f818 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain. Labels: Inferred, Low ###### The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12,… Id: claim_1582628a796f38cb1aa6f44b Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements. Labels: Contested, High ###### Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires. Id: claim_1c8e5efb564db325461219ed Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires. Labels: Inferred, Medium ###### Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need". Id: claim_2351ca4124f8b5ca525d28b4 Parent: claim_109b2b65de9358845145e539 Depth: 5 Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need". Labels: Inferred, Low ###### The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research… Id: claim_2905404d3f41f7914360eace Parent: claim_3b462da67628dea1b18ad066 Depth: 5 The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research organization structure around 2016–2017. Labels: Inferred, Medium ###### Aidan Gomez's post-paper trajectory corroborates the "intern" framing. Id: claim_29b7f293e56db098c430cfd9 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's post-paper trajectory corroborates the "intern" framing. Labels: Source-linked, Low ###### Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, s… Id: claim_2ce24a09545c8a197dc2e75d Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, shared with Kaiser. Labels: Inferred, Low ###### After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher. Id: claim_2ef74462624f17a1105b7885 Parent: claim_109b2b65de9358845145e539 Depth: 5 After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher. Labels: Source-linked, Low ###### Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017. Id: claim_3b0c2b5650bede0a8d5b9793 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017. Labels: Inferred, Low ###### Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google. Id: claim_40a8b51324a07ffac3abd1fd Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google. Labels: Inferred, Medium ###### The contribution statement separately assigns the design and implementation of the first Transformer models t… Id: claim_4b10c80ed716e7b07f329150 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer. Labels: Contested, High ###### Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@goog… Id: claim_5360e74850a613b537dcdfa9 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@google.com. Labels: Source-linked, Medium ###### Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain. Id: claim_627f8468d869594aad311db2 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain. Labels: Source-linked, Low ###### Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswan… Id: claim_6c2f25f51fdc506e74eae9ef Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswani@google.com. Labels: Source-linked, Medium ###### Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain. Id: claim_6f7aaf6af412bef85f08099c Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain. Labels: Inferred, Medium ###### Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appear… Id: claim_7329262d557cb3c88da6502d Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appeared in June 2017. Labels: Source-linked, Low ###### In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in… Id: claim_7e21aad5fc8968b329c9459f Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness. Labels: Contested, High ###### Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advan… Id: claim_82da7a91cd09f744e66bf490 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advance human understanding.” Labels: Inferred, Medium ###### Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a univer… Id: claim_879b1092306147a9b6c53f69 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a university rather than Google Brain or Google Research. Labels: Source-linked, Low ###### The finished Transformer model did not literally replace every RNN operation with attention and was not compo… Id: claim_9bcb2a40fd7529f5d29441cb Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention. Labels: Contested, High ###### Aidan Gomez was mentored by Łukasz Kaiser at Google Brain. Id: claim_9c9e077cd131dcbb7d81303a Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was mentored by Łukasz Kaiser at Google Brain. Labels: Inferred, Low ###### Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work perf… Id: claim_a5d9d87e695550c4173e7e6b Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work performed while at Google Research.” Labels: Source-linked, Medium ###### Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage intern… Id: claim_a7d566a2f6a4788226bbb2cb Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage internship task. Labels: Inferred, Low ###### Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natu… Id: claim_c2bd06aeb23fc5e699a37031 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natural-language inference and was trivially parallelizable. Labels: Inferred, High ###### Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receivin… Id: claim_cebce0e0fd2071c866cfca51 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receiving credit. Labels: Inferred, High ###### The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran. Id: claim_d3152341d9c9e78ad12f1863 Parent: claim_109b2b65de9358845145e539 Depth: 5 The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran. Labels: Inferred, Low ###### Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer… Id: claim_d68ea69fb5a7c1512fcd27d7 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer project crystallized in early 2017. Labels: Inferred, Low ###### Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA… Id: claim_d974d47be68e901004343515 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA team) on the initial Transformer designs. Labels: Inferred, Medium ###### Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attentio… Id: claim_e55cf2199bd87da2d9705d39 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attention Is All You Need" paper. Labels: Source-linked, Medium ###### The added statement of Uszkoreit's credit, published by the full author group under equal-contribution author… Id: claim_ef7f401d9060e217a4d7b656 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution. Labels: Contested, High ###### Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July. Id: claim_fe133f878247b5aa365b2a27 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July. Labels: Inferred, Medium ###### The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypoth… Id: claim_ff061eb3d5f7c5f5753f0034 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer. Labels: Contested, High