# Did TPU Hardware Economics, Not Algorithmic Superiority, Drive the Transformer's Rise?: 119 Sourced Claims, Contested Facts & Evidence The Transformer succeeded primarily because its operations matched dense TPU matrix architecture, not through algorithmic superiority. Authors: nineDan HTML: https://moyomap.com/maps/did-tpu-hardware-economics Markdown: https://moyomap.com/maps/did-tpu-hardware-economics.md License: HTML and markdown are for reading and citation. Bulk copying and AI training need a written license. See /terms and /developers#maps-data. Cite a claim as: MoyoMap, "Did TPU Hardware Economics, Not Algorithmic Superiority, Drive the Transformer's Rise?: 119 Sourced Claims, Contested Facts & Evidence", claim {id}, https://moyomap.com/maps/did-tpu-hardware-economics.md#{id}. “Moyo, a product of SenTeGuard Inc.” ### Hardware Determinism over Algorithmic Elegance Id: claim_daa701db06f0a02e21720724 Parent: topic_71F03FTXRXe6PHDp6azP Depth: 1 The Transformer architecture traded algorithmic sample efficiency for hardware-level training parallelizability on TPU clusters. Labels: Inferred, High, investigate ### Attention Is All You Need paper details Id: section_3bb0f6ddaccd61d270a97258 Parent: topic_71F03FTXRXe6PHDp6azP Depth: 1 Attention Is All You Need paper details Labels: Inferred, Low ### Prior work on attention in NMT Id: section_678cb7f2e9d079a0f7423515 Parent: topic_71F03FTXRXe6PHDp6azP Depth: 1 Prior work on attention in NMT Labels: Inferred, Low ### Preceding convolutional sequence models Id: section_e3f89c305de932b096e7d6f5 Parent: topic_71F03FTXRXe6PHDp6azP Depth: 1 Preceding convolutional sequence models Labels: Inferred, Low ### Arguments for Transformer architecture Id: section_e63c905fa4d2338e5c698455 Parent: topic_71F03FTXRXe6PHDp6azP Depth: 1 Arguments for Transformer architecture Labels: Inferred, Low #### The empirical exploration experiments for the Popel and Bojar (2018) Transformer training tips paper required… Id: claim_01566c39d255a2775ae70b46 Parent: claim_daa701db06f0a02e21720724 Depth: 2 The empirical exploration experiments for the Popel and Bojar (2018) Transformer training tips paper required roughly 4 years of cumulative GPU compute time. Labels: Source-linked, Low #### On an 11GB GPU, the Transformer BIG model ran out of memory (OOM) at batch sizes of 3000 and above, whereas t… Id: claim_0aba6e930a23a69c7c5ed722 Parent: claim_daa701db06f0a02e21720724 Depth: 2 On an 11GB GPU, the Transformer BIG model ran out of memory (OOM) at batch sizes of 3000 and above, whereas the BASE model operated up to batch size 6000. Labels: Source-linked, Low #### The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs preven… Id: claim_104b5d55a4d80dddbeb75b47 Parent: section_e63c905fa4d2338e5c698455 Depth: 2 The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples. Labels: Inferred, Low, investigate #### Scaling Transformer training from 1 GPU to 6 GPUs yielded a 3.2x throughput increase, whereas scaling batch s… Id: claim_23c18a7230cd454a57e68a5b Parent: claim_daa701db06f0a02e21720724 Depth: 2 Scaling Transformer training from 1 GPU to 6 GPUs yielded a 3.2x throughput increase, whereas scaling batch size from 1000 to 6000 on a single GPU yielded only a 1.3x increase. Labels: Inferred, Low #### The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NI… Id: claim_25d22bc9b022d66e6d9fb46b Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California. Labels: Inferred, Low, investigate #### Training-cost FLOP figures in the original Transformer paper were calculated as wall-clock time multiplied by… Id: claim_2facba4cd05924575dc7b8f1 Parent: claim_daa701db06f0a02e21720724 Depth: 2 Training-cost FLOP figures in the original Transformer paper were calculated as wall-clock time multiplied by GPU count and assumed sustained throughput (9.5 TFLOPS per P100), rather than measured operation counts. Labels: Source-linked, Low #### Strubell et al. Id: claim_328731bfcb4b53629d86655d Parent: claim_daa701db06f0a02e21720724 Depth: 2 Strubell et al. report the Transformer big model as having 213 million parameters, differing from the 165 million parameter count reported in the NeurIPS proceedings abstract. Labels: Contested, Low #### The Evolved Transformer architecture search required 32,623 TPU hours on a TPUv2 core, equivalent to roughly… Id: claim_40986ddb32aebe1b213ea587 Parent: claim_daa701db06f0a02e21720724 Depth: 2 The Evolved Transformer architecture search required 32,623 TPU hours on a TPUv2 core, equivalent to roughly 274,120 hours on 8 P100 GPUs. Labels: Source-linked, Low #### The "Attention Is All You Need" paper had 8 authors. Id: claim_4335e804701eb2d27e59d83c Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper had 8 authors. Labels: Inferred, Low, known #### The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent Engli… Id: claim_45f72b45492f4c5c69385fcb Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures. Labels: Source-linked, High #### The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2… Id: claim_4a6f33f21d8d28e30780bdf8 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023. Labels: Source-linked, High #### The Transformer big model achieved 28.4 EN-DE BLEU at an estimated 2.3 × 10¹⁹ FLOPs compared to ConvS2S Ensem… Id: claim_4cba9a489d7cc79d68a73548 Parent: claim_daa701db06f0a02e21720724 Depth: 2 The Transformer big model achieved 28.4 EN-DE BLEU at an estimated 2.3 × 10¹⁹ FLOPs compared to ConvS2S Ensemble's 26.36 BLEU at 7.7 × 10¹⁹ FLOPs. Labels: Inferred, Low #### Convolutional sequence models like ConvS2S already possessed O(1) sequential operations per layer, matching s… Id: claim_4ee4f7d328fbb82855c7e3b9 Parent: claim_daa701db06f0a02e21720724 Depth: 2 Convolutional sequence models like ConvS2S already possessed O(1) sequential operations per layer, matching self-attention's sequential operation complexity. Labels: Inferred, Low #### Melis, Dyer & Blunsom demonstrated in July 2017 that standard regularized LSTM architectures outperform newer… Id: claim_4f055f03238c9a0225f8586a Parent: claim_daa701db06f0a02e21720724 Depth: 2 Melis, Dyer & Blunsom demonstrated in July 2017 that standard regularized LSTM architectures outperform newer models when given equal hyperparameter tuning budgets. Labels: Source-linked, Low #### Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already… Id: claim_50a7a769868fc4f59a99cbba Parent: section_e3f89c305de932b096e7d6f5 Depth: 2 Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working. Labels: Inferred, Low, investigate #### In controlled low-resource NMT experiments, an optimized Transformer only outperforms an optimized RNN when t… Id: claim_70b67c1e2fe06401f3d16cba Parent: claim_daa701db06f0a02e21720724 Depth: 2 In controlled low-resource NMT experiments, an optimized Transformer only outperforms an optimized RNN when training data exceeds approximately 20,000 sentence pairs. Labels: Source-linked, Low #### Transformer training throughput exhibits sublinear growth with batch size because certain GPU operations cann… Id: claim_70de5fa395c00e8fad4701f7 Parent: claim_daa701db06f0a02e21720724 Depth: 2 Transformer training throughput exhibits sublinear growth with batch size because certain GPU operations cannot be fully batch-parallelized. Labels: Source-linked, Low #### The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, w… Id: claim_78c4d03acfda5c1e431b4342 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, with individual contributions listed. Labels: Inferred, Low, investigate #### Per Table 1 of the 2017 paper, self-attention exhibits per-layer computational complexity of O(n²·d) versus O… Id: claim_91c59918515c54f68fbb6b9a Parent: claim_daa701db06f0a02e21720724 Depth: 2 Per Table 1 of the 2017 paper, self-attention exhibits per-layer computational complexity of O(n²·d) versus O(n·d²) for recurrent layers, making self-attention cheaper only when sequence length n is less than dimension d. Labels: Inferred, Low #### The base Transformer model consumed 3.3 × 10¹⁸ FLOPs on English-German translation, while an English-French m… Id: claim_a5cb8d59bfe4f87bc44b8dcc Parent: claim_daa701db06f0a02e21720724 Depth: 2 The base Transformer model consumed 3.3 × 10¹⁸ FLOPs on English-German translation, while an English-French mixture-of-experts model required 1.2 × 10²⁰ FLOPs. Labels: Inferred, Low #### Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is A… Id: claim_a5dd3020d70fe88e48fcc606 Parent: section_678cb7f2e9d079a0f7423515 Depth: 2 Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is All You Need" paper. Labels: Inferred, Low, investigate #### Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the curr… Id: claim_aad33d84b1d247813216ddc3 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones. Labels: Source-linked, High #### The headline experiments in the 2017 'Attention Is All You Need' paper were conducted on 8 NVIDIA P100 GPUs o… Id: claim_ad64a6c085235357437e6e00 Parent: claim_daa701db06f0a02e21720724 Depth: 2 The headline experiments in the 2017 'Attention Is All You Need' paper were conducted on 8 NVIDIA P100 GPUs on a single machine rather than TPU clusters. Labels: Inferred, Low #### An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and foun… Id: claim_b80e9ab8420d41de82379488 Parent: section_3bb0f6ddaccd61d270a97258 Depth: 2 An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential. Labels: Source-linked, High #### There are no identified contemporaneous 2017 public sources empirically quantifying Transformer autoregressiv… Id: claim_db2a6dc2e0147cea3f9703eb Parent: claim_daa701db06f0a02e21720724 Depth: 2 There are no identified contemporaneous 2017 public sources empirically quantifying Transformer autoregressive decoding KV-cache memory overhead against recurrent neural network constant decode state. Labels: Inferred, Low #### The Mogrifier LSTM achieved state-of-the-art results on all tested language modeling datasets except Enwik8,… Id: claim_ef467fa9ca990be6f9a80b3d Parent: claim_daa701db06f0a02e21720724 Depth: 2 The Mogrifier LSTM achieved state-of-the-art results on all tested language modeling datasets except Enwik8, where it closed the gap against Transformer models. Labels: Source-linked, Low #### Transformers require larger batch sizes even under low-resource training conditions, whereas RNNs achieve sub… Id: claim_fd27c44b49e3116fd587f602 Parent: claim_daa701db06f0a02e21720724 Depth: 2 Transformers require larger batch sizes even under low-resource training conditions, whereas RNNs achieve substantial gains from small batch sizes. Labels: Source-linked, Low ##### Noam Shazeer is credited with creating scaled dot-product attention mechanisms. Id: claim_0d47febb0e537915283262de Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Noam Shazeer is credited with creating scaled dot-product attention mechanisms. Labels: Inferred, Low ##### The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of… Id: claim_10507064ebfbd9415ef97676 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8. Labels: Source-linked, High ##### The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs). Id: claim_139d1b7fea36382e7b721bef Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs). Labels: Inferred, Low ##### The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing. Id: claim_185da04e4ac0d53c5b99082a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing. Labels: Inferred, Low ##### The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Us… Id: claim_2421b88eeac3865c33c09b83 Parent: claim_4335e804701eb2d27e59d83c Depth: 3 The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Labels: Inferred, Low, known ##### The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models. Id: claim_25af15c7ff66517d6c26ab05 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models. Labels: Inferred, Low ##### Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks. Id: claim_25fd22118b2a09e17b031857 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks. Labels: Inferred, Low ##### The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12,… Id: claim_3a4651bba13bb101f5eeda23 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12, 2017. Labels: Source-linked, Low ##### The "Attention Is All You Need" paper introduced the Transformer architecture. Id: claim_3a7d50c24ff46a8d85088538 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper introduced the Transformer architecture. Labels: Inferred, Low ##### The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) confe… Id: claim_41a4844534fd7c4b90870bcb Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) conference. Labels: Source-linked, Low ##### The path length between distant positions in RNNs grew as O(n). Id: claim_46a5d61c940bc61b5a396996 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The path length between distant positions in RNNs grew as O(n). Labels: Inferred, Low, investigate ##### The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS. Id: claim_48881a625dea2017f74a9c70 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS. Labels: Source-linked, Low ##### The Transformer architecture allowed for more efficient computation on modern hardware. Id: claim_4a36eddcc067c80e84a38897 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer architecture allowed for more efficient computation on modern hardware. Labels: Inferred, Low ##### The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualif… Id: claim_4d76cb4d9c8147188efeeaa8 Parent: claim_b80e9ab8420d41de82379488 Depth: 3 The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualifying narratives that attribute results primarily to the attention architecture. Labels: Inferred, High ##### Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of enc… Id: claim_519d256a1fb1ad0a5daa1fbf Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of encoder–decoder RNNs. Labels: Inferred, Low, investigate ##### Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural mach… Id: claim_51a8f842952cc360ff383900 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural machine translation systems. Labels: Inferred, Low ##### Convolutional sequence models already had the same intra-example sequential-operation count as the models dis… Id: claim_527d67cee884c4d3c3a3f84d Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 Convolutional sequence models already had the same intra-example sequential-operation count as the models discussed in the context of the "Attention Is All You Need" paper. Labels: Inferred, Low ##### The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Lo… Id: claim_53c446607316ffd6cf7d6647 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Long Beach, California. Labels: Source-linked, Low ##### The author order in the "Attention Is All You Need" paper is noted to be random. Id: claim_5f4852fb0a40531df43ef49a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The author order in the "Attention Is All You Need" paper is noted to be random. Labels: Inferred, Low ##### The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015. Id: claim_7838b53c87fbbfb557a525cb Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015. Labels: Source-linked, Low ##### The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scor… Id: claim_8671fb9b2f5425721bd07edf Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU. Labels: Source-linked, High ##### Individual contributions of authors are specified in the "Attention Is All You Need" paper. Id: claim_927a2631eb7f38c8ef98a855 Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Individual contributions of authors are specified in the "Attention Is All You Need" paper. Labels: Inferred, Low ##### By 2016–2017, attention was a standard component in competitive translation systems, always used in conjuncti… Id: claim_955e920110f242d05c55c30d Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 By 2016–2017, attention was a standard component in competitive translation systems, always used in conjunction with recurrence. Labels: Inferred, Low, investigate ##### ConvS2S / Fairseq, by Gehring et al. Id: claim_99035a6869c842fbf8099f84 Parent: claim_50a7a769868fc4f59a99cbba Depth: 3 ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model. Labels: Inferred, Low, investigate ##### The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1. Id: claim_9debcc10bf59d8bd5da6fdc3 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1. Labels: Source-linked, High ##### ByteNet, by Kalchbrenner et al. Id: claim_a8d8ea70456a26d0592b06fa Parent: claim_50a7a769868fc4f59a99cbba Depth: 3 ByteNet, by Kalchbrenner et al. in 2016, used dilated convolutions and had logarithmic path length. Labels: Inferred, Low, investigate ##### The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from… Id: claim_b9a83bf140d0eddeb258e8ee Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from existing methods. Labels: Inferred, Low ##### The Transformer architecture eliminated the need for recurrence. Id: claim_bfedc33e72975b4a27fa4b0a Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer architecture eliminated the need for recurrence. Labels: Inferred, Low ##### Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score o… Id: claim_c15150c72de3031520b6f3d8 Parent: claim_45f72b45492f4c5c69385fcb Depth: 3 Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0. Labels: Source-linked, High ##### The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural… Id: claim_cca93f2326999722b1467da0 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural premise rather than a measured experiment. Labels: Inferred, Low ##### The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that… Id: claim_ccd26dae58615e121a02e0eb Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples, tracks the paper's own Introduction almost word-for-word. Labels: Inferred, Low ##### The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014. Id: claim_cf7b5744004331163fcf1803 Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014. Labels: Source-linked, Low ##### The "Attention Is All You Need" paper introduced the concept of self-attention. Id: claim_d57e202f2dc33ba69f8da2ec Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The "Attention Is All You Need" paper introduced the concept of self-attention. Labels: Inferred, Low ##### The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of thre… Id: claim_d712b01269c19f2eb9b11095 Parent: claim_104b5d55a4d80dddbeb75b47 Depth: 3 The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of three co-equal arguments used by the authors of "Attention Is All You Need." Labels: Inferred, Low ##### Noam Shazeer is credited with creating multi-head attention mechanisms. Id: claim_d784640207e1bc9ebb338b3f Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Noam Shazeer is credited with creating multi-head attention mechanisms. Labels: Inferred, Low ##### The latest official version of "Attention is All you Need" on arXiv is v7, not v5. Id: claim_d92dbf535e9d17e84f12dbd7 Parent: claim_4a6f33f21d8d28e30780bdf8 Depth: 3 The latest official version of "Attention is All you Need" on arXiv is v7, not v5. Labels: Source-linked, High ##### Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model. Id: claim_dafa5deb89d2537da9ecc9dc Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model. Labels: Inferred, Low ##### Luong et al. Id: claim_dd8d5477211f558e777b61e2 Parent: claim_a5dd3020d70fe88e48fcc606 Depth: 3 Luong et al. refined attention with multiplicative/dot-product attention and local attention in 2015. Labels: Inferred, Low, investigate ##### A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperpar… Id: claim_e5cfcd8c8338eb60c1c582d7 Parent: claim_b80e9ab8420d41de82379488 Depth: 3 A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperparameters, had to be extracted from the code rather than the submitted paper. Labels: Source-linked, High ##### The Transformer model enables faster and more effective training through parallelization. Id: claim_f79c6824562cf7ef914326ce Parent: claim_25d22bc9b022d66e6d9fb46b Depth: 3 The Transformer model enables faster and more effective training through parallelization. Labels: Inferred, Low ###### Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Ne… Id: claim_0d2c97b07d6f43a44fc37581 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper. Labels: Inferred, Low, investigate ###### Aidan Gomez was an intern from the University of Toronto when the paper was published. Id: claim_109b2b65de9358845145e539 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Aidan Gomez was an intern from the University of Toronto when the paper was published. Labels: Inferred, Low, investigate ###### Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research. Id: claim_3b462da67628dea1b18ad066 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research. Labels: Inferred, Low, investigate ###### Niki Parmar is credited with designing, implementing, and evaluating model variants. Id: claim_5dbc904100ca44aa24c53b26 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Niki Parmar is credited with designing, implementing, and evaluating model variants. Labels: Inferred, Low, investigate ###### Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase. Id: claim_6d0097192c0a222447405477 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase. Labels: Inferred, Low, investigate ###### The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/b… Id: claim_812a02b718b83b27da1a8639 Parent: claim_8671fb9b2f5425721bd07edf Depth: 4 The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy. Labels: Source-linked, High ###### Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models. Id: claim_9ca1153e396783334d865203 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models. Labels: Inferred, Low, investigate ###### Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free posi… Id: claim_9d4b882fbeaed19302ec9237 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation. Labels: Inferred, Low, investigate ###### Llion Jones is credited with visualizations and efficient inference work. Id: claim_b102d8f4e92a851157df7923 Parent: claim_2421b88eeac3865c33c09b83 Depth: 4 Llion Jones is credited with visualizations and efficient inference work. Labels: Inferred, Low, investigate ###### The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revisi… Id: claim_da3643812bb1e6f193eedc49 Parent: claim_10507064ebfbd9415ef97676 Depth: 4 The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revision, not in the original posting. Labels: Inferred, High ###### Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the Universi… Id: claim_094f0cc9e843ff87fb2b7137 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the University of Toronto. Labels: Inferred, Low ###### The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole ar… Id: claim_0aa591ea5d4c6243a596698e Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor. Labels: Contested, High ###### Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszka… Id: claim_0b98ca1e8cf109d0371bd8f5 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszkaiser@google.com. Labels: Source-linked, Medium ###### Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You… Id: claim_0e4a5e8e5d71048234414eb6 Parent: claim_109b2b65de9358845145e539 Depth: 5 Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You Need" paper. Labels: Inferred, Low ###### Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2… Id: claim_10f7df62feefb096e46c5130 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2016 discussion. Labels: Inferred, High ###### Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain. Id: claim_139d1d390cfb472dbbe1f818 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain. Labels: Inferred, Low ###### The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12,… Id: claim_1582628a796f38cb1aa6f44b Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements. Labels: Contested, High ###### Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires. Id: claim_1c8e5efb564db325461219ed Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires. Labels: Inferred, Medium ###### Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need". Id: claim_2351ca4124f8b5ca525d28b4 Parent: claim_109b2b65de9358845145e539 Depth: 5 Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need". Labels: Inferred, Low ###### The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research… Id: claim_2905404d3f41f7914360eace Parent: claim_3b462da67628dea1b18ad066 Depth: 5 The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research organization structure around 2016–2017. Labels: Inferred, Medium ###### Aidan Gomez's post-paper trajectory corroborates the "intern" framing. Id: claim_29b7f293e56db098c430cfd9 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's post-paper trajectory corroborates the "intern" framing. Labels: Source-linked, Low ###### Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, s… Id: claim_2ce24a09545c8a197dc2e75d Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, shared with Kaiser. Labels: Inferred, Low ###### After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher. Id: claim_2ef74462624f17a1105b7885 Parent: claim_109b2b65de9358845145e539 Depth: 5 After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher. Labels: Source-linked, Low ###### Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017. Id: claim_3b0c2b5650bede0a8d5b9793 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017. Labels: Inferred, Low ###### Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google. Id: claim_40a8b51324a07ffac3abd1fd Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google. Labels: Inferred, Medium ###### The contribution statement separately assigns the design and implementation of the first Transformer models t… Id: claim_4b10c80ed716e7b07f329150 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer. Labels: Contested, High ###### Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@goog… Id: claim_5360e74850a613b537dcdfa9 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@google.com. Labels: Source-linked, Medium ###### Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain. Id: claim_627f8468d869594aad311db2 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain. Labels: Source-linked, Low ###### Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswan… Id: claim_6c2f25f51fdc506e74eae9ef Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswani@google.com. Labels: Source-linked, Medium ###### Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain. Id: claim_6f7aaf6af412bef85f08099c Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain. Labels: Inferred, Medium ###### Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appear… Id: claim_7329262d557cb3c88da6502d Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appeared in June 2017. Labels: Source-linked, Low ###### In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in… Id: claim_7e21aad5fc8968b329c9459f Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness. Labels: Contested, High ###### Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advan… Id: claim_82da7a91cd09f744e66bf490 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advance human understanding.” Labels: Inferred, Medium ###### Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a univer… Id: claim_879b1092306147a9b6c53f69 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a university rather than Google Brain or Google Research. Labels: Source-linked, Low ###### The finished Transformer model did not literally replace every RNN operation with attention and was not compo… Id: claim_9bcb2a40fd7529f5d29441cb Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention. Labels: Contested, High ###### Aidan Gomez was mentored by Łukasz Kaiser at Google Brain. Id: claim_9c9e077cd131dcbb7d81303a Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez was mentored by Łukasz Kaiser at Google Brain. Labels: Inferred, Low ###### Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work perf… Id: claim_a5d9d87e695550c4173e7e6b Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work performed while at Google Research.” Labels: Source-linked, Medium ###### Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage intern… Id: claim_a7d566a2f6a4788226bbb2cb Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage internship task. Labels: Inferred, Low ###### Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natu… Id: claim_c2bd06aeb23fc5e699a37031 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natural-language inference and was trivially parallelizable. Labels: Inferred, High ###### Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receivin… Id: claim_cebce0e0fd2071c866cfca51 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receiving credit. Labels: Inferred, High ###### The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran. Id: claim_d3152341d9c9e78ad12f1863 Parent: claim_109b2b65de9358845145e539 Depth: 5 The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran. Labels: Inferred, Low ###### Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer… Id: claim_d68ea69fb5a7c1512fcd27d7 Parent: claim_109b2b65de9358845145e539 Depth: 5 Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer project crystallized in early 2017. Labels: Inferred, Low ###### Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA… Id: claim_d974d47be68e901004343515 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA team) on the initial Transformer designs. Labels: Inferred, Medium ###### Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attentio… Id: claim_e55cf2199bd87da2d9705d39 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attention Is All You Need" paper. Labels: Source-linked, Medium ###### The added statement of Uszkoreit's credit, published by the full author group under equal-contribution author… Id: claim_ef7f401d9060e217a4d7b656 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution. Labels: Contested, High ###### Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July. Id: claim_fe133f878247b5aa365b2a27 Parent: claim_3b462da67628dea1b18ad066 Depth: 5 Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July. Labels: Inferred, Medium ###### The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypoth… Id: claim_ff061eb3d5f7c5f5753f0034 Parent: claim_0d2c97b07d6f43a44fc37581 Depth: 5 The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer. Labels: Contested, High