“But Google is the author of ‘Attention Is All You Need’, they have an advantage” – Not really

I hear a lot that Google is the inventor of the foundation for LLM's: The transformer model. Therefore, they have an advantage. I disagree, here is why:

ChatGPT and all other LLM's rely heavily on the neural network building block called Transformer. It consists of MultiHeadAttention layers, therefore the name of the paper “Attention Is All You Need”. A transformer model allows to input the sequences in parallel. It is remarkable, as pervious sequence to sequence models (as input text, to response text is) required going through the input sequence word by word in sequence. So transformer models are so much cheaper and faster to train, which allows them to grow much quicker in size.

So now that we know this, how do we built on this knowledge? You would assume that Google, the inventor of this transformer architecture, would know best. However, the knowledge is with the authors. They know how and why they came up with what they came up with. And they know the behaviour best and how to build on top to create large, awesome language models. So lets see what these authors are working on right now:

• Ashish Vaswani: Co-founder of Stealth Startup
• Noam Shazeer: CEO Character.AI
• Niki Parmar: Co-founder of Stealth Startup
• Jakob Uszkoreit: Co-founder of Inceptive
• Llion Jones: Director of Canolfan Bedwyr
• Aidan N. Gomez: Co-founder of Cohere
• Lukasz Kaiser: Researcher at OpenAI and CNRS
• Illia Polosukhin: Co-founder of NEAR

Except for Illia (working on blockhain technology) all are working on LLM's, but none at Google. Lukasz even works at OpenAI now.

My point is, even though Google has been the “inventor” of Transformer Models and therefore LLM's, there has been a huge knowledge outflow. Now you can argue about the reasons for this. But without the authors, Google can only do what everyone else does: Reading the papers and trying to learn on how to use it. Google does not have an advantage, and I doubt they have something up their sleeves.

There are many search queries, that will still be answered best by Google, no doubt for the near future. But all queries that require chain of thought, combining information, or transferring knowledge will best be dealt by LLM's. And here the best LLM will win, which I doubt is being developed by Google right now.

(Disclaimer: This was not written by chatGPT. Current work of authors was helped with by bing.)