Cross-Attention
A highly specialized attention mechanism where the queries come directly from the decoder, while the keys and values are explicitly pulled from the encoder.
Think of It Like This
Like a chef (decoder) constantly looking back at a specific recipe card (encoder) to figure out exactly how much salt they need to add to the soup next.
Unlike self-attention, which looks strictly within its own sequence to find context, cross-attention acts as a bridge between two entirely different sequences. It is the critical architectural component in Seq2Seq models like T5, allowing the model generating a French translation to constantly look back at the original English text for the exact right word.