Transformer Neural Network

In neural network architectures that manage data like text or signals, transformers are heavily utilized. They are frequently found in language processing scenarios. In these environments, transformers take a sequence of vectors as an input and convert it into a different type of vector known as an encoding. Significantly, the transformer involves a key component named the attention mechanism, which influences the encoding of a token depending on the relevance of other tokens in the input. This mechanism enables the transformer to focus on specific words surrounding the target word to facilitate a proper translation. In the sphere of deep learning, transformer neural networks are progressively replacing long-established neural network configurations like RNN, LSTM, and GRU.

Transformer Neural Network model

The transformer model in machine-learning essentially transforms a sentence into two distinct threads: vector embeddings and positional encodings. Word vectors, or numerical representatives, numerically exhibit the text. For neural networks to manage these words, they need to be converted into an embedding format. Here, words take the form of vectors. In positional encodings, the position of the word is represented in vector format.

Post the addition of word embeddings and encodings, the sum is redirected through tiers of encoders, subsequently routed through tiers of decoders. The transformer takes a different route from RNNs and LSTMs since it submits the entire input at once, as against RNNs and LSTMs, which inject the entire input in a sequential pattern.

Discussing transformation, encoders generate encodings by converting the input into another vector series, a process reversed during decoding. In the decoding stage, the encoded-words are recalculated into probabilities of other output words, which are then translated into another natural language sentence using the softmax function.

TNN vs RNN

Transformers differ fundamentally from RNNs in layout. An RNN consistently maintains a concealed state vector. Any input word is circulated through a neural network’s tiers, tweaking the state vector. However, the model's concealed state usually fails to retain substantial information about preliminary inputs, making it vulnerable to data loss with new inputs. Furthermore, the sequential processing of input sequences makes RNNs incompatible with computational tools like GPUs.

TNN vs LSTM

On the other hand, the cell state forms the backbone of the LSTM structure. Special structures called “gates” control the modification of information in a hidden cell state in LSTM, which uses RNN structure as its basis. These gates partially resolve the issue of long-term dependency in LSTM. Still, sequential processing is necessary for LSTM training and operation, making parallel computation difficult and increasing training time.

Transformer networks have an edge over both LSTMs and RNNs in that they allow for simultaneous processing of multiple words. Though numerous studies demonstrated improved LSTM performance using attention mechanisms before the advent of transformer architecture, researchers eventually found that a simple attention mechanism could replace the need for a recurrent neural network, leading to the birth of the parallel-structured transformer, which facilitates training on graphics processing units.