Research summary
Research summary
Topic summary
Far from attention being all that was needed, feed-forward layers perform the vast majority of factual computation and parameter storage.
Key research questions
- Organize claims about claims into a claim tree.
- Investigate this contrarian claim about Attention is All you Need.
Important claims and supporting citations
The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8.
The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.
The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures.
The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023.
The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy.
The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU.
The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.
Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones.
Competing explanations
The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.
The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor.
The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer.
In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness.
The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention.
The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution.
The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer.
Outstanding questions
- The 'Attention' Misnomer and MLP Dominance
- Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper.
- The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples.
- Aidan Gomez was an intern from the University of Toronto when the paper was published.
- The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California.
- Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research.
- The path length between distant positions in RNNs grew as O(n).
- Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working.
Lihat semua klaim (teks)
Klaim
Claim text is for reading and citation. Bulk copying and AI training need a licensed maps data API key. Markdown
The 'Attention' Misnomer and MLP Dominance
Feed-forward sublayers function as associative key-value memories that store the substantive knowledge in Transformer models.
Labels: Inferred, High, investigate
Attention Is All You Need paper details
Attention Is All You Need paper details
Labels: Inferred, Low
Prior work on attention in NMT
Prior work on attention in NMT
Labels: Inferred, Low
Preceding convolutional sequence models
Preceding convolutional sequence models
Labels: Inferred, Low
Arguments for Transformer architecture
Arguments for Transformer architecture
Labels: Inferred, Low
ROME's key-value memory formulation relies on classical linear associative memory frameworks originally estab…
ROME's key-value memory formulation relies on classical linear associative memory frameworks originally established by Anderson (1972) and Kohonen (1972).
Labels: Inferred, Low
The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs preven…
The "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples.
Labels: Inferred, Low, investigate
Under ROME's mechanistic model, summed subject information accumulated by MLPs is copied to the last token by…
Under ROME's mechanistic model, summed subject information accumulated by MLPs is copied to the last token by attention mechanisms at high layers.
Labels: Inferred, Low
The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NI…
The "Attention Is All You Need" paper was published on June 12, 2017, as arXiv:1706.03762 and presented at NIPS 2017, which took place from December 4–9, 2017, in Long Beach, California.
Labels: Inferred, Low, investigate
Associative-memory localization and ROME model editing techniques transfer successfully to Mamba, a non-trans…
Associative-memory localization and ROME model editing techniques transfer successfully to Mamba, a non-transformer state-space model.
Labels: Inferred, Low
The original implementation of Causal Tracing in ROME restores 10 layers simultaneously, indicating that caus…
The original implementation of Causal Tracing in ROME restores 10 layers simultaneously, indicating that causal signals are spread across multiple consecutive MLPs rather than isolated to a single layer.
Labels: Inferred, Low
The "Attention Is All You Need" paper had 8 authors.
The "Attention Is All You Need" paper had 8 authors.
Labels: Inferred, Low, known
The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent Engli…
The public record of the "Attention is All you Need" paper contains at least four mutually inconsistent English–French BLEU figures.
Labels: Source-linked, High
The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2…
The arXiv record for "Attention is All you Need" includes versions v6 on July 24, 2023, and v7 on August 2, 2023.
Labels: Source-linked, High
Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already…
Immediately preceding work showed that recurrence was not necessary, with convolutional alternatives already working.
Labels: Inferred, Low, investigate
Geva et al.
Geva et al. (2023) show that subject representations are enriched with attributes by early MLP sublayers, while attribute extraction is carried out by attention heads that encode subject-attribute mappings in their parameters.
Labels: Inferred, Low
Hase et al.
Hase et al. found that representation zeroing yields near-zero correlations with edit success across layers, matching the non-predictiveness found with representation denoising.
Labels: Inferred, Low
In model editing experiments by Hase et al., layer choice accounts for 94.7% of the variance in rewrite score…
In model editing experiments by Hase et al., layer choice accounts for 94.7% of the variance in rewrite score, while adding causal tracing effects increases R² by only 0.1% to 94.8%.
Labels: Inferred, Low
Specific attention heads directly encode subject-attribute mappings in 30.2% of cases in GPT-2 and 39.3% in G…
Specific attention heads directly encode subject-attribute mappings in 30.2% of cases in GPT-2 and 39.3% in GPT-J, with extraction behavior observed in roughly 70% of predictions.
Labels: Inferred, Low
The key-value associative memory framing of transformer FFNs draws directly on Sukhbaatar et al.
The key-value associative memory framing of transformer FFNs draws directly on Sukhbaatar et al. (2015) End-to-End Memory Networks.
Labels: Inferred, Low
The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, w…
The "Attention Is All You Need" paper includes an unusual footnote stating that the author order is random, with individual contributions listed.
Labels: Inferred, Low, investigate
Causal intervention in ROME localizes factual association specifically to middle-layer feed-forward modules w…
Causal intervention in ROME localizes factual association specifically to middle-layer feed-forward modules while processing the subject's last token.
Labels: Inferred, Low
Hase et al.
Hase et al. (2023) demonstrated that causal localization via representation denoising (Causal Tracing) is statistically uncorrelated with the success of fact-injecting edits across MLP layers.
Labels: Inferred, Low
Lv et al.
Lv et al. found that task-specific attention heads move topic entities to the final position while MLPs execute the relation function.
Labels: Inferred, Low
Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is A…
Attention was already a load-bearing component in Neural Machine Translation (NMT) before the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate
Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the curr…
Ashish Vaswani was identified as the submitter for v1 of "Attention is All you Need" on arXiv, while the current record identifies Llion Jones.
Labels: Source-linked, High
An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and foun…
An independent NeurIPS reviewer attempted to reimplement the system from "Attention is All you Need" and found the learning-rate schedule to be unusually consequential.
Labels: Source-linked, High
Knocking out early MLP sublayers in autoregressive models leads to up to an approximately 88% reduction in th…
Knocking out early MLP sublayers in autoregressive models leads to up to an approximately 88% reduction in the attributes rate of subject representations.
Labels: Inferred, Low
In the Fact Forcing variant tested by Hase et al., causal tracing results correlate with edit success, though…
In the Fact Forcing variant tested by Hase et al., causal tracing results correlate with edit success, though selecting early-to-mid-layer MLP weights without tracing remains superior.
Labels: Inferred, Low
In transformer feed-forward layers, shallow layers only detect shallow sentence patterns, whereas upper layer…
In transformer feed-forward layers, shallow layers only detect shallow sentence patterns, whereas upper layers encode semantic content.
Labels: Inferred, Low
Chen et al.
Chen et al. found that multiple distinct sets of knowledge neurons can store identical facts rather than facts being isolated to a single MLP location.
Labels: Inferred, Low
Geva et al.
Geva et al. (2021) state that feed-forward layers constitute two-thirds of a transformer model's parameters and operate as key-value memories where keys match training-corpus textual patterns and values induce output-vocabulary distributions.
Labels: Inferred, Low
Feed-forward layer outputs in transformers are compositions of multiple memories refined across layers rather…
Feed-forward layer outputs in transformers are compositions of multiple memories refined across layers rather than single-memory lookups.
Labels: Inferred, Low
Associations edited by ROME are directional, meaning reverse relational queries (e.g., subject-object inversi…
Associations edited by ROME are directional, meaning reverse relational queries (e.g., subject-object inversion) require independent edits.
Labels: Inferred, Low
Pre-edit facts survive inside later-layer attention heads within specific left singular vectors of the attent…
Pre-edit facts survive inside later-layer attention heads within specific left singular vectors of the attention output matrix after MLP editing (including ROME, MEMIT, and AlphaEdit) and can be recovered by ablating those vectors.
Labels: Inferred, Low
Noam Shazeer is credited with creating scaled dot-product attention mechanisms.
Noam Shazeer is credited with creating scaled dot-product attention mechanisms.
Labels: Inferred, Low
The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of…
The current arXiv abstract and Table 2 for "Attention is All you Need" report an English–French BLEU score of 41.8.
Labels: Source-linked, High
The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs).
The Transformer model contrasts with the sequential nature of Recurrent Neural Networks (RNNs).
Labels: Inferred, Low
The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing.
The Transformer model, introduced in "Attention Is All You Need," revolutionized natural language processing.
Labels: Inferred, Low
The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Us…
The authors of the "Attention Is All You Need" paper were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.
Labels: Inferred, Low, known
The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models.
The "Attention Is All You Need" paper directly influenced the development of various Transformer-based models.
Labels: Inferred, Low
Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks.
Before "Attention Is All You Need," attention mechanisms were typically combined with recurrent networks.
Labels: Inferred, Low
The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12,…
The paper "Attention Is All You Need" was published on arXiv with the identifier arXiv:1706.03762 on June 12, 2017.
Labels: Source-linked, Low
The "Attention Is All You Need" paper introduced the Transformer architecture.
The "Attention Is All You Need" paper introduced the Transformer architecture.
Labels: Inferred, Low
The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) confe…
The paper "Attention Is All You Need" was presented at the Neural Information Processing Systems (NIPS) conference.
Labels: Source-linked, Low
The path length between distant positions in RNNs grew as O(n).
The path length between distant positions in RNNs grew as O(n).
Labels: Inferred, Low, investigate
The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS.
The NIPS conference where "Attention Is All You Need" was presented is now known as NeurIPS.
Labels: Source-linked, Low
The Transformer architecture allowed for more efficient computation on modern hardware.
The Transformer architecture allowed for more efficient computation on modern hardware.
Labels: Inferred, Low
The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualif…
The optimizer schedule was an apparently load-bearing implementation detail for the Transformer model, qualifying narratives that attribute results primarily to the attention architecture.
Labels: Inferred, High
Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of enc…
Bahdanau, Cho, and Bengio introduced additive attention in 2014 to address the fixed-length bottleneck of encoder–decoder RNNs.
Labels: Inferred, Low, investigate
Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural mach…
Prior to the "Attention Is All You Need" paper, attention mechanisms were a critical component of neural machine translation systems.
Labels: Inferred, Low
Convolutional sequence models already had the same intra-example sequential-operation count as the models dis…
Convolutional sequence models already had the same intra-example sequential-operation count as the models discussed in the context of the "Attention Is All You Need" paper.
Labels: Inferred, Low
The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Lo…
The NIPS conference where "Attention Is All You Need" was presented took place from December 4–9, 2017, in Long Beach, California.
Labels: Source-linked, Low
The author order in the "Attention Is All You Need" paper is noted to be random.
The author order in the "Attention Is All You Need" paper is noted to be random.
Labels: Inferred, Low
The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015.
The v1 version of the Luong, Pham & Manning paper (arXiv:1508.04025) appeared in August 2015.
Labels: Source-linked, Low
The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scor…
The NeurIPS proceedings abstract for "Attention is All you Need" describes a 165-million-parameter model scoring 27.5 English–German BLEU.
Labels: Source-linked, High
Individual contributions of authors are specified in the "Attention Is All You Need" paper.
Individual contributions of authors are specified in the "Attention Is All You Need" paper.
Labels: Inferred, Low
By 2016–2017, attention was a standard component in competitive translation systems, always used in conjuncti…
By 2016–2017, attention was a standard component in competitive translation systems, always used in conjunction with recurrence.
Labels: Inferred, Low, investigate
ConvS2S / Fairseq, by Gehring et al.
ConvS2S / Fairseq, by Gehring et al. in May 2017 from Facebook AI Research, was a fully convolutional seq2seq model.
Labels: Inferred, Low, investigate
The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.
The NeurIPS proceedings abstract for "Attention is All you Need" reports an English–French BLEU score of 41.1.
Labels: Source-linked, High
ByteNet, by Kalchbrenner et al.
ByteNet, by Kalchbrenner et al. in 2016, used dilated convolutions and had logarithmic path length.
Labels: Inferred, Low, investigate
The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from…
The "Attention Is All You Need" paper's proposal to use self-attention alone was a significant departure from existing methods.
Labels: Inferred, Low
The Transformer architecture eliminated the need for recurrence.
The Transformer architecture eliminated the need for recurrence.
Labels: Inferred, Low
Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score o…
Table 2 of the NeurIPS proceedings PDF for "Attention is All you Need" reports an English–French BLEU score of 41.0.
Labels: Source-linked, High
The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural…
The selected claim, regarding sequential computation in RNNs preventing parallelization, is an architectural premise rather than a measured experiment.
Labels: Inferred, Low
The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that…
The selected claim, which states that the "Attention Is All You Need" paper's core empirical argument is that sequential computation in RNNs prevents parallelization within training examples, tracks the paper's own Introduction almost word-for-word.
Labels: Inferred, Low
The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014.
The Bahdanau, Cho & Bengio paper (arXiv:1409.0473) was first submitted as v1 on September 1, 2014.
Labels: Source-linked, Low
The "Attention Is All You Need" paper introduced the concept of self-attention.
The "Attention Is All You Need" paper introduced the concept of self-attention.
Labels: Inferred, Low
The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of thre…
The selected claim, concerning sequential computation in RNNs preventing parallelization, is only one of three co-equal arguments used by the authors of "Attention Is All You Need."
Labels: Inferred, Low
Noam Shazeer is credited with creating multi-head attention mechanisms.
Noam Shazeer is credited with creating multi-head attention mechanisms.
Labels: Inferred, Low
The latest official version of "Attention is All you Need" on arXiv is v7, not v5.
The latest official version of "Attention is All you Need" on arXiv is v7, not v5.
Labels: Source-linked, High
Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model.
Scaled dot-product attention and multi-head attention mechanisms are key innovations in the Transformer model.
Labels: Inferred, Low
Luong et al.
Luong et al. refined attention with multiplicative/dot-product attention and local attention in 2015.
Labels: Inferred, Low, investigate
A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperpar…
A reviewer of "Attention is All you Need" found that important replication details, such as numerous hyperparameters, had to be extracted from the code rather than the submitted paper.
Labels: Source-linked, High
The Transformer model enables faster and more effective training through parallelization.
The Transformer model enables faster and more effective training through parallelization.
Labels: Inferred, Low
Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Ne…
Jakob Uszkoreit is credited with proposing replacing RNNs with self-attention in the "Attention Is All You Need" paper.
Labels: Inferred, Low, investigate
Aidan Gomez was an intern from the University of Toronto when the paper was published.
Aidan Gomez was an intern from the University of Toronto when the paper was published.
Labels: Inferred, Low, investigate
Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research.
Ashish Vaswani, Noam Shazeer, Łukasz Kaiser, and Illia Polosukhin were affiliated with Google Brain/Research.
Labels: Inferred, Low, investigate
Niki Parmar is credited with designing, implementing, and evaluating model variants.
Niki Parmar is credited with designing, implementing, and evaluating model variants.
Labels: Inferred, Low, investigate
Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase.
Aidan Gomez and Łukasz Kaiser are credited with the tensor2tensor codebase.
Labels: Inferred, Low, investigate
The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/b…
The proceedings PDF body of "Attention is All you Need" uses 28.4 English–German BLEU, creating an abstract/body discrepancy.
Labels: Source-linked, High
Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models.
Ashish Vaswani and Illia Polosukhin are credited with designing and implementing the first Transformer models.
Labels: Inferred, Low, investigate
Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free posi…
Noam Shazeer is credited with scaled dot-product attention, multi-head attention, and the parameter-free position representation.
Labels: Inferred, Low, investigate
Llion Jones is credited with visualizations and efficient inference work.
Llion Jones is credited with visualizations and efficient inference work.
Labels: Inferred, Low, investigate
The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revisi…
The headline 41.8 English–French BLEU result for "Attention is All you Need" was introduced in a later revision, not in the original posting.
Labels: Inferred, High
Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the Universi…
Aidan Gomez's path to the "Attention Is All You Need" paper traces to Geoffrey Hinton's group at the University of Toronto.
Labels: Inferred, Low
The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole ar…
The best-supported interpretation of Uszkoreit's role is originator and initial project catalyst, not sole architect or inventor.
Labels: Contested, High
Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszka…
Łukasz Kaiser was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email lukaszkaiser@google.com.
Labels: Source-linked, Medium
Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You…
Tensor2tensor was open-sourced by Google, announced in June 2017, the same month as the "Attention Is All You Need" paper.
Labels: Inferred, Low
Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2…
Illia Polosukhin recalled Uszkoreit suggesting self-attention for Google Search question answering during a 2016 discussion.
Labels: Inferred, High
Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain.
Aidan Gomez performed the work for "Attention Is All You Need" as an intern at Google Brain.
Labels: Inferred, Low
The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12,…
The explicit sentence attributing the proposal to Uszkoreit does not appear in arXiv versions v1–v4 (June 12, 19, 20, 30, 2017) but appears in v5 (December 6, 2017) with detailed contribution statements.
Labels: Contested, High
Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires.
Noam Shazeer was a Google employee since approximately 2000, making him one of the earliest hires.
Labels: Inferred, Medium
Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need".
Łukasz Kaiser was a Google Research Poland-based coauthor of "Attention Is All You Need".
Labels: Inferred, Low
The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research…
The distinction between “Google Brain” and “Google Research” in the paper reflects Google’s internal research organization structure around 2016–2017.
Labels: Inferred, Medium
Aidan Gomez's post-paper trajectory corroborates the "intern" framing.
Aidan Gomez's post-paper trajectory corroborates the "intern" framing.
Labels: Source-linked, Low
Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, s…
Aidan Gomez's specific credited contribution to "Attention Is All You Need" was the tensor2tensor codebase, shared with Kaiser.
Labels: Inferred, Low
After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher.
After the "Attention Is All You Need" paper, Aidan Gomez spent time at Google Brain as a researcher.
Labels: Source-linked, Low
Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017.
Aidan Gomez was roughly 20 years old when "Attention Is All You Need" appeared in June 2017.
Labels: Inferred, Low
Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google.
Illia Polosukhin co-founded the project that later became NEAR Protocol after leaving Google.
Labels: Inferred, Medium
The contribution statement separately assigns the design and implementation of the first Transformer models t…
The contribution statement separately assigns the design and implementation of the first Transformer models to Ashish Vaswani and Illia Polosukhin, and several defining mechanisms to Noam Shazeer.
Labels: Contested, High
Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@goog…
Noam Shazeer was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email noam@google.com.
Labels: Source-linked, Medium
Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain.
Aidan Gomez had a stint with Geoffrey Hinton's Toronto group while at Google Brain.
Labels: Source-linked, Low
Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswan…
Ashish Vaswani was listed as being affiliated with “Google Brain” in arXiv:1706.03762, with the email avaswani@google.com.
Labels: Source-linked, Medium
Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain.
Ashish Vaswani was based in Mountain View (Building 1965) during his time at Google Brain.
Labels: Inferred, Medium
Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appear…
Aidan Gomez was an undergraduate student at the University of Toronto when "Attention Is All You Need" appeared in June 2017.
Labels: Source-linked, Low
In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in…
In an October 2024 interview, Uszkoreit identified his principal contribution as proposing that recurrence in sequence-transduction models could be replaced by self-attention, hypothesizing increased efficiency and effectiveness.
Labels: Contested, High
Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advan…
Ashish Vaswani described Google Brain as “a radical group” that believed “neural networks were going to advance human understanding.”
Labels: Inferred, Medium
Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a univer…
Aidan Gomez is the only one of the eight authors of "Attention Is All You Need" whose affiliation is a university rather than Google Brain or Google Research.
Labels: Source-linked, Low
The finished Transformer model did not literally replace every RNN operation with attention and was not compo…
The finished Transformer model did not literally replace every RNN operation with attention and was not composed solely of attention.
Labels: Contested, High
Aidan Gomez was mentored by Łukasz Kaiser at Google Brain.
Aidan Gomez was mentored by Łukasz Kaiser at Google Brain.
Labels: Inferred, Low
Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work perf…
Illia Polosukhin's affiliation in arXiv:1706.03762 was clarified by an explicit footnote stating “‡ Work performed while at Google Research.”
Labels: Source-linked, Medium
Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage intern…
Aidan Gomez's involvement in the Transformer project spans its formative period, not just a late-stage internship task.
Labels: Inferred, Low
Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natu…
Uszkoreit was a coauthor of the Decomposable Attention Model by June 2016, which used attention to split natural-language inference and was trivially parallelizable.
Labels: Inferred, High
Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receivin…
Uszkoreit's first-person testimony corroborates the attribution but is not independent of the person receiving credit.
Labels: Inferred, High
The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran.
The tensor2tensor codebase was the infrastructure on which the Transformer experiments ran.
Labels: Inferred, Low
Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer…
Aidan Gomez reportedly began working with Google researchers (via Kaiser) around 2016, before the Transformer project crystallized in early 2017.
Labels: Inferred, Low
Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA…
Ashish Vaswani collaborated closely with Illia Polosukhin (who was on the adjacent Building 1945 language/QA team) on the initial Transformer designs.
Labels: Inferred, Medium
Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attentio…
Illia Polosukhin's departure from Google occurred before the June 12, 2017, arXiv v1 posting of the "Attention Is All You Need" paper.
Labels: Source-linked, Medium
The added statement of Uszkoreit's credit, published by the full author group under equal-contribution author…
The added statement of Uszkoreit's credit, published by the full author group under equal-contribution authorship, is an important provenance qualification but does not undermine the attribution.
Labels: Contested, High
Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July.
Ashish Vaswani joined Google Brain as a research scientist in mid-2016, specifically around July.
Labels: Inferred, Medium
The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypoth…
The phrase 'proposed replacing RNNs with self-attention' describes the project's initial architectural hypothesis, not the sole invention of the complete Transformer.
Labels: Contested, High